Nothing Special   »   [go: up one dir, main page]

WO2020215952A1 - 物品识别方法和系统 - Google Patents

物品识别方法和系统 Download PDF

Info

Publication number
WO2020215952A1
WO2020215952A1 PCT/CN2020/080767 CN2020080767W WO2020215952A1 WO 2020215952 A1 WO2020215952 A1 WO 2020215952A1 CN 2020080767 W CN2020080767 W CN 2020080767W WO 2020215952 A1 WO2020215952 A1 WO 2020215952A1
Authority
WO
WIPO (PCT)
Prior art keywords
item
image
category
recognized
information
Prior art date
Application number
PCT/CN2020/080767
Other languages
English (en)
French (fr)
Inventor
马事伟
吴江旭
张伟华
石海龙
张洪光
徐荣图
胡淼枫
王璟璟
Original Assignee
北京京东尚科信息技术有限公司
北京京东世纪贸易有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京京东尚科信息技术有限公司, 北京京东世纪贸易有限公司 filed Critical 北京京东尚科信息技术有限公司
Publication of WO2020215952A1 publication Critical patent/WO2020215952A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the present disclosure relates to the field of image recognition, and in particular to an item recognition method and system.
  • the dishes need to be identified first, and then settled according to the prices corresponding to the dishes.
  • dishes can be identified based on computer vision technology.
  • the sensor triggers the image acquisition device to take a picture of the dish, and then recognizes the dish; or, processes each image collected by the image acquisition device to identify the dish information in the image.
  • an item identification method which includes: acquiring one or more images to be identified, wherein the images to be identified include one or more items to be identified; using a trained pre-recognition model to determine Identify whether the probability of the image being clear and containing the complete item is greater than a threshold; and if the probability is greater than the threshold, identify the category of each item.
  • the probability of the first image being clear and containing the complete item is greater than the first threshold, and the probability of other images being clear and containing the complete item is greater than the second threshold, the The items contained in the first image among the images are classified.
  • training the pre-recognition model includes: labeling images in the sample images that are clear and containing complete items as positive sample images, and labeling images that are not positive sample images in the sample images as negative sample images; and The sample image and the negative sample image train the pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than the threshold according to the trained pre-recognition model.
  • identifying the category of each item includes: inputting the image to be recognized into the item detection model, extracting the area information and the first level category corresponding to each item in the image to be recognized; determining the effective category in the first level category; And input the area information of the items belonging to the valid category in the image to be recognized into the item recognition model, extract the item features corresponding to each area information, and compare the item features corresponding to each area information with the item features in the item feature library to determine The second-level category of each item in the image to be recognized.
  • training the item detection model and the item recognition model includes: labeling the area information and the first-level category corresponding to the item in the sample image, generating first label information, and labeling the item based on the sample image and the first label information.
  • the detection model is trained to determine the area information and the first-level category corresponding to each item in the image to be recognized according to the trained item detection model; and to label the item features corresponding to the area information of the effective category items in the sample image
  • the second tagging information is to train the item recognition model based on the sample image and the second tagging information, so as to extract the item features corresponding to the area information of each item in the image to be recognized based on the trained item recognition model.
  • the minimum distance between the item feature of each item and the item feature in the item feature library is determined; if the minimum distance is less than or equal to the distance threshold, the item feature corresponding to each item in the item feature library is the closest to the item feature
  • the corresponding category is regarded as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether the item category and attribute information needs to be input; and if the item category and attribute need to be input, the item category and attribute information is added, otherwise ,
  • the category corresponding to the closest item feature in the item feature library corresponding to each item is taken as the second level category of each item.
  • the corresponding attribute information is matched according to the category of each item.
  • the image to be recognized is marked as a training image or a test image, so that the item detection model and the item recognition model are performed based on the image to be recognized. Training or testing.
  • the size information of each item is determined based on the item detection model, and the corresponding attribute information is matched according to the category and size of each item; it is determined whether multiple items in the image to be identified meet the item combination, if multiple items meet the item combination , Match the attribute information corresponding to the item combination; determine whether the attribute sum corresponding to multiple items in the image to be identified meets the preset condition, if the attribute sum satisfies the preset condition, the attribute sum is processed according to the preset condition; and determine For the matching time of item matching attribute information, the attribute information corresponding to each item is determined according to the matching time.
  • an item identification system including: an image acquisition module configured to acquire one or more images to be identified, wherein the images to be identified include one or more items to be identified;
  • the pre-recognition module is configured to use the trained pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than a threshold; and the item determination module is configured to recognize the probability of each item when the probability is greater than the threshold category.
  • the pre-recognition module is further configured to: if the probability of the first image being clear and containing the complete item is greater than the first threshold, and the probability of other images being clear and containing the complete item is greater than the first If the threshold is two, the first image in the consecutive multiple images is sent to the item determination module; and the item determination module is configured to perform category recognition on the items contained in the first image in the consecutive multiple images.
  • the pre-recognition module is further configured to mark images in the sample image that are clear and contain complete items as positive sample images, and to mark images in the sample images that are not positive sample images as negative sample images; and
  • the positive sample image and the negative sample image train the pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than the threshold according to the trained pre-recognition model.
  • the item determination module includes: an item detection module configured to input the image to be recognized into the item detection model, and extract the area information and the first-level category corresponding to each item in the image to be recognized based on the item detection model;
  • the management module is configured to determine the effective category in the first-level category;
  • the item recognition module is configured to input the area information of items belonging to the effective category in the image to be recognized into the item recognition model, and extract each area based on the item recognition model The item features corresponding to the information are compared with the item features in the item feature database to determine the second-level category of each item in the image to be recognized.
  • the item management module is configured to determine the effective category in the first-level category; the item detection module is configured to input the image to be recognized into the item detection model, and extract the area corresponding to each item in the image to be recognized Information and the first-level category, call the item management module, and input the area information of items belonging to the effective category to the item recognition module; and the item recognition module is also configured to identify the item characteristics corresponding to the area information of the effective category items in the sample image Annotation is performed to generate second annotation information, and the item recognition model is trained based on the sample image and the second annotation information, so as to extract the item features corresponding to the area information of each item in the image to be recognized according to the trained item identification model.
  • the item management module is configured to determine valid item features in the item feature library within a predetermined time; and the item identification module is further configured to compare the item features corresponding to each area information with the valid item features in the item feature library. Perform comparison to determine the second-level category of each item in the image to be identified.
  • the item identification module is configured to determine the minimum distance between the item feature of each item and the item feature in the item feature library; if the minimum distance is less than or equal to the distance threshold, the item in the item feature library corresponding to each item The category corresponding to the item feature with the closest feature distance is regarded as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether the item category and attribute information needs to be input; if the item category and attribute need to be input, the item category is added And attribute information, otherwise, the category corresponding to the item feature closest to the item feature corresponding to each item in the item feature library is taken as the second level category of each item.
  • the attribute matching unit is configured to match corresponding attribute information according to the category of each item.
  • the item management module is further configured to, after matching the attribute information, in response to the user modifying the attribute information corresponding to the category of the item, mark the image to be recognized as a training image or a test image, so as to compare the item based on the image to be recognized.
  • the detection model and the item recognition model are trained or tested.
  • the attribute matching unit is further configured to: match corresponding attribute information according to the category and size of each item, wherein the item detection module is further configured to determine the size information of each item based on the item detection model ; Determine whether multiple items in the image to be recognized meet the item combination, if multiple items meet the item combination, match the attribute information corresponding to the item combination; determine whether the attribute sum corresponding to multiple items in the image to be recognized meets the preset conditions, If the preset conditions are met, the attributes are processed according to the preset conditions; and the matching time of the item matching attribute information is determined, and the attribute information corresponding to each item is determined according to the matching time.
  • an article identification system including: a memory; and a processor coupled to the memory, the processor configured to execute the above-mentioned article identification method based on instructions stored in the memory.
  • a non-transitory computer-readable storage medium on which computer program instructions are stored, and when the instructions are executed by a processor, the above-mentioned item identification method is realized.
  • FIG. 1 is a schematic flowchart of some embodiments of the object identification method of the present disclosure.
  • FIG. 2 is a schematic flowchart of other embodiments of the object identification method of the present disclosure.
  • FIG. 3 is a schematic structural diagram of some embodiments of the object identification system of the present disclosure.
  • Fig. 4 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 5 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 6 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 7 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 8 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 1 is a schematic flowchart of some embodiments of the object identification method of the present disclosure.
  • one or more images to be identified are acquired, where the images to be identified include one or more objects to be identified.
  • the images to be identified include one or more objects to be identified. For example, in a restaurant, when a customer purchases a dish and a bowl of rice, the dish and rice can be placed in the recognition area, and the recognition area can be photographed by a camera to obtain an image containing the dish and rice.
  • step 120 the trained pre-recognition model is used to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • the pre-recognition model may be trained in advance, a certain number of sample images are collected, and the sample images are classified. For example, the image in the sample image is clear and the image containing the complete item is annotated as a positive sample image, the image in the sample image that is not a positive sample image is annotated as a negative sample image, and the pre-recognition model is performed based on the positive sample image and the negative sample image. training.
  • the output result of the pre-recognition module is compared with the corresponding information of the sample to determine whether the comparison result meets the requirements of the loss function of the pre-recognition model. Iteratively, optimize and adjust the parameters of the pre-recognition module, so that the comparison result finally meets the requirements of constructing the pre-recognition model The requirement of the loss function, save the pre-recognition model.
  • a customer purchases a dish and a bowl of rice, first places the dish and rice on a tray, and then places the tray in the recognition area.
  • the tray is constantly moving. If the tray does not completely enter the recognition area when the image capture device captures an image, the tray in the image is incomplete. For example, some dishes are not collected, or some dishes are collected only a small part, which will affect the accuracy of subsequent identification. In addition, when the tray is moving, the collected images are blurred with motion, which will also affect the accuracy of subsequent recognition. Therefore, invalid images are excluded, and dishes are recognized only for images that contain a complete tray and have clear images.
  • a certain number of images of the recognition area are collected, including images of no tray in the area, images of the tray just entering the recognition area, images of half of the tray entering the recognition area, and images of the tray completely entering the recognition area. Then classify the images, label the images with clear images and the trays in the image completely entering the recognition area as positive sample images, label other images as negative sample images, and then train the pre-recognition model based on the positive and negative sample images. If the customer does not use the tray, the image is clear, and the images containing the complete dishes, beverages and other priced goods are marked as positive sample images, and the other images are marked as negative sample images, and then the pre-recognition model is trained.
  • step 130 when the probability is greater than the threshold, the category of each item is identified. That is, in this embodiment, instead of performing item category recognition on all images, it first determines whether the image meets the requirements, and performs item recognition on the images that meet the requirements.
  • each item in the image whose image is clear and the probability of containing a complete item is greater than the threshold is recognized, instead of performing item recognition on all images, the accuracy and recognition efficiency of the recognition system can be improved.
  • the probability of each image being clear and containing a complete item in multiple consecutive images to be recognized is greater than the threshold, then the first image contained in the consecutive multiple images Items are identified by category.
  • the probability that the first image is clear and contains a complete item is greater than the first threshold, and the other images are clear and contain complete If the probability of the item is greater than the second threshold, the item contained in the first image among the consecutive multiple images is classified.
  • the item contained in the first image is identified.
  • the second to Nth images are clear and the probability of containing a complete object is greater than 0.1, the second to Nth images are not processed.
  • FIG. 2 is a schematic flowchart of other embodiments of the object identification method of the present disclosure.
  • step 210 one or more images to be identified are acquired, where the images to be identified include one or more items to be identified.
  • step 220 the trained pre-recognition model is used to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • step 230 when the probability is greater than the threshold, the image to be recognized is input to the item detection model, and the region information and the first level category corresponding to each item in the image to be recognized are extracted based on the item detection model.
  • the first-level category refers to the category to which the item belongs, such as dishes, fruits, and beverages.
  • the item detection model can be pre-trained, the area information and the first-level category corresponding to the item in the sample image are annotated, the first annotation information is generated, and the item detection model is performed based on the sample image and the first annotation information. training. Compare the output result of the item detection model with the first label information to determine whether the comparison result meets the requirements of the loss function for building the item recognition model, and iterate repeatedly, optimize and adjust the parameters of the item detection model, so that the comparison result finally meets the construction of the item detection model The requirements of the loss function, save the item detection model.
  • the item detection model When training the item detection model, mark the items in the collected images as categories such as dishes, yogurt, fruits, drinks, keys, badges, wallets, mobile phones, chopsticks, spoons, and hands. Then input the image to the item detection model for training. After the item detection model is trained, when an image is input, the item detection model can output the area information and category information of each item in the image.
  • a valid category in the first-level category is determined according to the configuration information. For example, for non-valuable items, there is a probability that they will be mistaken as valued items. Therefore, it is necessary to remove the category of non-valued items and only keep the category of valued items to avoid misidentification.
  • step 250 the area information of the items belonging to the effective category in the image to be recognized is input into the item recognition model, and the item features corresponding to each area information are extracted based on the item recognition model, and the item features corresponding to each area information are compared with those in the item feature database.
  • the item features are compared to determine the second-level category of each item in the image to be recognized.
  • the second level category can correspond to the specific information of the item. For example, a certain dish is specifically fried green peppers or fried cabbage.
  • the item features corresponding to the area information of the effective category items in the sample image are annotated, the second annotation information is generated, and the item recognition model is trained based on the sample image and the second annotation information. Compare the output result of the item recognition model with the second label information, determine whether the comparison result meets the requirements of the loss function of building the item recognition model, iterate repeatedly, optimize and adjust the parameters of the item recognition model, so that the comparison result finally meets the construction of the item recognition model The requirements of the loss function, save the item recognition model.
  • an image of the dish is first collected, and the image is input to the item detection module, and the item detection model outputs the area information and category of the dish. Then, the dish feature corresponding to the regional information of the dish is annotated, and the image and the annotation information are input to the item recognition model to train the item recognition model.
  • the item recognition model calls the feature library, compares the output dish features with the dish features stored in the feature library, and recognizes the The specific information corresponding to the dish, for example, whether the dish is fried cabbage or fried green pepper.
  • the image is pre-identified first, the unqualified images are removed, and then the categories of the items in the images that meet the requirements are identified.
  • the invalid category is removed, and only the item features corresponding to the area information of the items belonging to the valid category are identified, and specific items can be identified based on the item features, which improves the accuracy of item identification.
  • the effective item features in the item feature database are determined within a predetermined time; the item features corresponding to each area information are compared with the effective item features in the item feature database to determine each The second level category of the item.
  • the item feature database saves the characteristics of each dish in each period, but the vegetables that make up a certain dish may be slightly different in different seasons, or, in certain periods, certain dishes are no longer on sale. Therefore, it is possible to set the features of dishes not currently participating in the sale as invalid features, and the features of dishes participating in the sale as valid features.
  • identifying a dish compare the characteristics of the dish to be identified with the characteristics of the effective dish in the feature library to determine the specific dish.
  • the item features corresponding to each area information are compared with the valid item features in the item feature library to determine the second-level category of each item in the image to be identified, which can reduce the interference in the item identification process, and further Improve the accuracy of recognition.
  • the item feature corresponding to each item in the item feature library is the category corresponding to the item feature closest to the item feature as the second level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether to enter the item category and attribute Information; if you need to enter the item category and attribute, add the item category and attribute information, otherwise, the item feature corresponding to each item in the item feature library is the category corresponding to the item feature closest to the item as the second level category of each item.
  • the distance is, for example, Euclidean distance, and the size of the distance represents the size of similarity. The smaller the distance, the more similar the item feature of the item to be identified and the item feature in the item feature library. When the distance exceeds the distance threshold, it means that the item feature database may not contain the feature of the item to be identified. Therefore, the user can be prompted whether to input item category and attribute information. If the user inputs, it means that a new item needs to be registered. If the user does not input, the category corresponding to the item feature closest to each item is taken as the second level category of each item.
  • the attribute information corresponding to the item is matched.
  • the attribute information is, for example, price. For example, after recognizing that a certain dish is stir-fried cabbage, the price corresponding to the dish can be matched. In the settlement, if there are multiple dishes, the multiple dishes can be settled.
  • the attribute information of the item can be more accurately matched.
  • the attribute information is price information, the accuracy of commodity settlement can be improved.
  • the image to be recognized is marked as a training image or a test image, so that the object detection model and the The item recognition model is trained or tested. For example, if it is recognized that a certain dish is stir-fried cabbage, and the price of stir-fried cabbage is matched, but in actual calculation, the user modifies the settlement price, it means that the dish was identified incorrectly. Therefore, the image containing the dish can be used as a training image or a test image, and the image can be used to train or test the object detection model and the object recognition model. Through automatic iteration of the model, the accuracy of the recognition of the model can be improved.
  • the size information of each item is determined based on the item detection model, and the corresponding attribute information is matched according to the category and size of each item.
  • the attribute information is the price.
  • the size boundary of the large and small dishes can be calculated, that is, the average value of the large dishes and the average value of the small dishes.
  • the size of the recognized dish is compared with the size boundary to determine whether the recognized dish is a large portion or a small portion, and then the corresponding price is matched.
  • the attribute information corresponding to the item combination is matched.
  • the set menu information is configured when the restaurant is settled. If you order 15 yuan for a single fried cabbage, 2 yuan for a bowl of rice, and 16 yuan for a fried cabbage and a rice, it is recognized that the image contains fried After cabbage and rice, you need to match the price of 16 yuan.
  • the matching time of item matching attribute information is determined, and the attribute information corresponding to each item is determined according to the matching time. For example, at the time of restaurant settlement, you can configure the discount period and the discount intensity to determine whether the time when the dish matches the price is in the discount period, and if so, you can match the dish with the discount price corresponding to the discount period.
  • FIG. 3 is a schematic structural diagram of some embodiments of the object identification system of the present disclosure.
  • the system includes an image acquisition module 310, a pre-identification module 320, and an item determination module 330.
  • the image acquisition module 310 is configured to acquire one or more images to be identified, where the images to be identified include one or more objects to be identified.
  • the pre-recognition module 320 is configured to use the trained pre-recognition model to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • the pre-recognition model may be trained in advance, a certain number of sample images are collected, and the sample images are classified. For example, the image in the sample image is clear and the image containing the complete item is annotated as a positive sample image, the image in the sample image that is not a positive sample image is annotated as a negative sample image, and the pre-recognition model is performed based on the positive sample image and the negative sample image. training.
  • the item determination module 330 is configured to identify the category of each item when the probability is greater than the threshold. That is, in this embodiment, instead of performing item category recognition on all images, it first determines whether the image meets the requirements, and performs item recognition on the images that meet the requirements.
  • each item in the image whose image is clear and the probability of containing a complete item is greater than the threshold is recognized instead of recognizing all the images, which can improve the accuracy and recognition efficiency of the recognition system.
  • the pre-recognition module 320 is further configured to, if the probability of the first image being clear and containing the complete item is greater than the first threshold, and the other images are clear and containing complete If the probability of the item is greater than the second threshold, the first image of the consecutive multiple images is sent to the item determination module 330.
  • the item determination module 330 is configured to perform category recognition on items contained in the first image of the consecutive plurality of images.
  • the second to Nth images are clear and the probability of containing a complete article is greater than 0.1, not processing the second to Nth images can reduce the processing burden of the article recognition system and improve system stability.
  • Fig. 4 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • the item determination module 330 in the system includes an item detection module 331, an item management module 332, and an item identification module 333.
  • the item detection module 331 is configured to input the image to be recognized into the item detection model, extract the area information and the first-level category corresponding to each item in the image to be recognized based on the item detection model, call the item management module 332, and set the items belonging to the valid category
  • the area information of the item is input to the item identification module 333.
  • the first-level category refers to the category to which the item belongs, such as dishes, fruits, and beverages.
  • the item detection model can be pre-trained, the area information and the first-level category corresponding to the item in the sample image are annotated, the first annotation information is generated, and the item detection model is performed based on the sample image and the first annotation information. training.
  • the item management module 332 is configured to determine the effective category in the first level category.
  • items such as dishes, yogurt, fruits, and beverages in the recognition area
  • non-valuable items such as keys, badges, wallets, mobile phones, chopsticks, spoons, and hands. Therefore, after the first-level category of the article is identified, the invalid category is removed first, and only the valid category is retained.
  • the item recognition module 333 is configured to input the area information of the items belonging to the valid category in the image to be recognized into the item recognition model, extract the item features corresponding to each area information based on the item recognition model, and combine the item features corresponding to each area information with the item features
  • the features of the items in the library are compared to determine the second-level category of each item in the image to be recognized.
  • the second level category can correspond to the specific information of the item. For example, a certain dish is specifically fried green peppers or fried cabbage.
  • the item features corresponding to the area information of the effective category items in the sample image are annotated, the second annotation information is generated, and the item recognition model is trained based on the sample image and the second annotation information.
  • the image is pre-identified first, the unqualified images are removed, and then the categories of the items in the images that meet the requirements are identified.
  • the invalid category is removed, and only the item features corresponding to the area information of the items belonging to the valid category are identified, and specific items can be identified based on the item features, which improves the accuracy of item identification.
  • the item management module 332 is further configured to determine valid item features in the item feature library within a predetermined time.
  • the item identification module 333 is also configured to compare the item features corresponding to each area information with the effective item features in the item feature library to determine the second-level category of each item in the map to be identified.
  • the item feature database saves the characteristics of each dish in each period, but the vegetables that make up a certain dish may be slightly different in different seasons, or, in certain periods, certain dishes are no longer on sale. Therefore, it is possible to set the features of dishes not currently participating in the sale as invalid features, and the features of dishes participating in the sale as valid features.
  • identifying a dish compare the characteristics of the dish to be identified with the characteristics of the effective dish in the feature library to determine the specific dish.
  • the item features corresponding to each area information are compared with the valid item features in the item feature library to determine the second-level category of each item in the image to be identified, which can reduce the interference in the item identification process, and further Improve the accuracy of recognition.
  • the item recognition module 333 is configured to determine the minimum distance between the item feature of each item and the item feature in the item feature library; if the minimum distance is less than or equal to the distance threshold, the item feature library and The item feature corresponding to each item is the category corresponding to the closest item feature as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether to enter the item category and attribute information; if the item category and attribute information need to be entered Attribute, add item category and attribute information, otherwise, the category corresponding to the item feature closest to the item feature corresponding to each item in the item feature library is taken as the second level category of each item.
  • the distance is, for example, Euclidean distance, and the size of the distance represents the size of similarity. The smaller the distance, the more similar the item feature of the item to be identified and the item feature in the item feature library. When the distance exceeds the distance threshold, it means that the item feature database may not contain the feature of the item to be identified. Therefore, the user can be prompted whether to input item category and attribute information. If the user inputs, it means that a new item needs to be registered. If the user does not input, the category corresponding to the item feature closest to each item is taken as the second level category of each item.
  • the system further includes an attribute matching unit 510 configured to match corresponding attribute information according to the category of each item.
  • the attribute information is, for example, price. For example, after recognizing that a certain dish is stir-fried cabbage, the price corresponding to the dish can be matched. In the settlement, if there are multiple dishes, the multiple dishes can be settled.
  • the attribute information of the item can be more accurately matched.
  • the attribute information is price information, the accuracy of commodity settlement can be improved.
  • the item management module 332 is further configured to, after matching the attribute information, in response to the user modifying the attribute information corresponding to the category of the item, mark the image to be recognized as a training image or a test image so as to be based on
  • the image to be recognized trains or tests the item detection model and the item recognition model. For example, if it is recognized that a certain dish is stir-fried cabbage, and the price of stir-fried cabbage is matched, but in actual calculation, the user modifies the settlement price, it means that the dish was identified incorrectly. Therefore, the image containing the dish can be used as a training image or a test image, and the image can be used to train or test the object detection model and the object recognition model. Through automatic iteration of the model, the accuracy of the recognition of the model can be improved.
  • the attribute matching unit 510 is further configured to match the corresponding attribute information according to the category and size of each item, wherein the item detection module 331 is further configured to determine the size of each item based on the item detection model.
  • the attribute information is the price.
  • the size boundary of the large and small dishes can be calculated, that is, the average value of the large dishes and the average value of the small dishes.
  • the size of the recognized dish is compared with the size boundary to determine whether the recognized dish is a large portion or a small portion, and then the corresponding price is matched.
  • the attribute matching unit 510 is further configured to determine whether multiple items in the image to be identified satisfy the item combination, and if the multiple items satisfy the item combination, match the attribute information corresponding to the item combination.
  • the attribute matching unit 510 is further configured to determine whether the attributes corresponding to multiple items in the image to be identified and whether they meet a preset condition, and if the preset condition is met, the attribute matching unit 510 is matched according to the preset condition. And processing.
  • the attribute matching unit 510 is further configured to determine the matching time of item matching attribute information, and determine the attribute information corresponding to each item according to the matching time.
  • this embodiment includes a registration module 610, a pre-identification module 620, an item detection module 630, an item identification module 640, an item management module 650, a search module 660, a feature library 670, and a settlement module 680.
  • the settlement module 680 corresponds to the attribute matching unit 510.
  • the registration module 610 calls the camera to collect images of the settlement area. For accurate subsequent identification, when registering priced commodities such as dishes and beverages, only one commodity is placed in the settlement area. For example, place only one plate of fried cabbage.
  • the registration module 610 inputs the image to the item detection module 630.
  • the item detection module 630 detects the area information of the item and sends the area information to the item identification module 640.
  • the item identification module 640 extracts the feature of the item and then stores the feature in the feature Library 670.
  • the product is brought to the checkout counter.
  • the settlement module 680 calls the camera to take an image
  • the camera sends the image to the pre-recognition module 620 to determine whether the image is available. That is, it is judged whether the probability that the image is clear and contains the complete product is greater than the threshold, and whether the image is the first image among consecutive images greater than the threshold, and if so, the pre-identification module 620 sends the image to the item detection module 630 .
  • the item detection module 630 detects the category of each item included in the image and outputs area information corresponding to each item.
  • the dish registration desk is usually placed inside the back kitchen to facilitate restaurant staff to register dishes.
  • the back kitchen is usually messy, and there may be some irrelevant things appearing near the registration desk. If non-dish items and other non-price items cannot be filtered, then the non-price items may be entered into the feature database, causing misunderstanding.
  • the collected images often include chopsticks, spoons, badges, mobile phones, wallets, hands and other items. These non-price items may be detected as dishes and cause misunderstanding. Calling the item management module 650 can remove non-valuable items and solve the problem of easy interference in commodity detection.
  • the item management module 650 can configure which types are non-priced items. For example, for example, some restaurants have drinks for sale, and some restaurants do not sell drinks, then the restaurant can configure whether drinks participate in the pricing according to the actual situation. For another example, if the restaurant has fruit delivery activities, you can configure the fruit not to participate in the pricing.
  • keys, badges, wallets, mobile phones, chopsticks, spoons, hands, etc. can be configured in the item management module 650 as non-valuable items.
  • the item management module 650 may also set the feature of the commodity not currently participating in the sale as an invalid feature. For example, the item management module 650 records the dishes and their prices sold at each time of day, and triggers menu synchronization through a timer.
  • the item management module 650 can also process daily order data, count the sales of dishes, and customer order information.
  • the item identification module 640 determines the feature information corresponding to the area information of the commodity in the pricing category, and calls the feature library 670 through the search module 660, and finds the feature closest to the feature of the commodity in the feature library 670, so that the item identification module 640 outputs the corresponding feature of the commodity Specific category, and send the product information to the settlement module 680.
  • a settlement system based on dish identification usually requires dish registration before the meal is opened.
  • some dishes such as temporary dishes, are only served after a certain period of time after the meal is opened. These temporary dishes cannot be registered before the meal is opened. Therefore, the temporary dishes cannot be identified at the time of settlement.
  • the user may be prompted, for example, whether the settlement clerk is registered for temporary dishes. If registered, the registration of the temporary dish is completed by entering the dish name and price, and the dish information is sent to the settlement module 680; if not, the dish information is sent to the settlement module 680 according to the current recognition result.
  • This embodiment can solve the problem that temporary dishes cannot be identified.
  • the settlement module 680 performs settlement according to the commodity category and price.
  • Some dishes in the restaurant have different sizes and sizes, and the prices for large and small portions are different.
  • the large portion of eight-treasure porridge is 6 yuan
  • the small portion is 3 yuan.
  • dishes of large and small portions are basically similar in appearance. Therefore, it is necessary to identify the size information of the dishes, and set the price of the large and small dishes in the settlement module 680.
  • the restaurant sells dishes, there may be set meal discount activities.
  • the unit price of clear soup ramen is 9 yuan
  • the unit price of beef slices is 9 yuan
  • the combination of clear soup ramen and beef slices is 16 yuan.
  • the package price needs to be set in the settlement module 680.
  • Some restaurants may discount certain dishes during certain time periods, for example, in the evening. Therefore, it is also necessary to configure the discount time period and discount strength in the settlement module 680. In some restaurants, there will be a full gift event. Therefore, the full gift information can also be set in the settlement module 680.
  • the accuracy of product identification is improved, the accuracy of product settlement can be improved, user experience can be improved, and the cost of product settlement can be reduced.
  • the system further includes an IoT (Internet of Things) platform 6100, an annotation platform 6110, and an algorithm server 6120.
  • IoT Internet of Things
  • the item management module 650 uploads the wrongly recognized image to the IoT platform 6100, the IoT platform 6100 submits the error data of the day to the labeling platform 6110, and the labeling platform 6100 returns the labelled data to the algorithm server 6120 after completing the labeling.
  • the algorithm server 6120 randomly divides the labeled data into a training set and a test set, and performs model training and model testing to improve model iteration efficiency. Before registering the product, the algorithm server 6120 trains each model in the product recognition process.
  • the registration module 610, the pre-identification module 620, and the settlement module 680 may be installed on the client; the item detection module 630, the item identification module 640, the item management module 650, the search module 660, and the feature library 670 may be installed on the server.
  • the module in the client can communicate with the module in the server through the service module 690; the IoT platform 6100, the annotation platform 6110, and the algorithm server 6120 can be set in the cloud.
  • FIG. 7 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • the system includes a memory 710 and a processor 720, where the memory 710 may be a magnetic disk, flash memory or any other non-volatile storage medium.
  • the memory is used to store instructions in the embodiments corresponding to FIGS. 1 and 2.
  • the processor 720 is coupled to the memory 710 and may be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller.
  • the processor 720 is configured to execute instructions stored in the memory.
  • the system 800 includes a memory 810 and a processor 820.
  • the processor 820 is coupled to the memory 810 through the BUS bus 830.
  • the system 800 can also be connected to an external storage device 850 through the storage interface 840 to call external data, and can also be connected to the network or another computer system (not shown) through the network interface 860, which will not be described in detail here.
  • the data instructions are stored in the memory, and the above instructions are processed by the processor, which can improve the accuracy of item identification.
  • a computer-readable storage medium has computer program instructions stored thereon, and when the instructions are executed by a processor, the steps of the method in the embodiments corresponding to FIGS. 1 and 2 are implemented.
  • the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.
  • the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. .
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

一种物品识别方法和系统，涉及图像识别领域。该方法包括：获取一个或多个待识别图像，其中，待识别图像中包括一个或多个待识别的物品(110)；利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值(120)；以及在概率大于阈值的情况下，识别各物品的类别(130)。该方法能够提高物品识别的准确性和效率。

Description

物品识别方法和系统
相关申请的交叉引用
本申请是以CN申请号为201910325808.X，申请日为2019年4月23日的申请为基础，并主张其优先权，该CN申请的公开内容在此作为整体引入本申请中。
技术领域
本公开涉及图像识别领域，尤其涉及一种物品识别方法和系统。
背景技术
在餐厅结算系统中，需要先识别出菜品，然后根据菜品对应的价格进行结算。相关技术中，可以基于计算机视觉技术识别菜品。例如，通过传感器触发图像采集装置对菜品进行拍照，然后进行菜品识别；或者，对图像采集装置采集的每一张图像进行处理，识别出图像中的菜品信息。
发明内容
根据本公开一方面，提出一种物品识别方法，包括：获取一个或多个待识别图像，其中，待识别图像中包括一个或多个待识别的物品；利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值；以及在概率大于阈值的情况下，识别各物品的类别。
在一些实施例中，若连续多个待识别图像中，第一个图像清晰并且包含完整物品的概率大于第一阈值，且其他图像清晰并且包含完整物品的概率大于第二阈值，则对连续多个图像中的第一个图像中包含的物品进行类别识别。
在一些实施例中，训练预识别模型包括：将样本图像中图像清晰并且包含完整物品的图像标注为正样本图像，将样本图像中不属于正样本图像的图像标注为负样本图像；以及基于正样本图像和负样本图像对预识别模型进行训练，以便根据训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值。
在一些实施例中，识别各物品的类别包括：将待识别图像输入至物品检测模型，提取待识别图像中各物品对应的区域信息和第一级类别；确定第一级类别中的有效类别；以及将待识别图像中属于有效类别的物品的区域信息输入至物品识别模型，提取 各区域信息对应的物品特征，将各区域信息对应的物品特征与物品特征库中的物品特征进行比对，确定待识别图像中各物品的第二级类别。
在一些实施例中，训练物品检测模型和物品识别模型包括：对样本图像中的物品对应的区域信息和第一级类别进行标注，生成第一标注信息，基于样本图像和第一标注信息对物品检测模型进行训练，以便根据训练好的物品检测模型，确定待识别图像中各物品对应的区域信息和第一级类别；以及对样本图像中的有效类别物品的区域信息对应的物品特征进行标注生成第二标注信息，基于样本图像和第二标注信息对物品识别模型进行训练，以便根据训练好的物品识别模型提取待识别图像中各物品的区域信息对应的物品特征。
在一些实施例中，确定预定时间内物品特征库中的有效物品特征；以及将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别。
在一些实施例中，确定各物品的物品特征与物品特征库中的物品特征的最小距离；若最小距离小于等于距离阈值，则将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别作为各物品的第二级类别；若最小距离大于距离阈值，则向用户提示是否需要输入物品类别和属性信息；以及若需要输入物品类别和属性，则增加物品类别和属性信息，否则，将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别作为各物品的第二级类别。
在一些实施例中，根据各物品的类别匹配对应的属性信息。
在一些实施例中，在匹配属性信息后，响应于用户修改物品的类别对应的属性信息，将待识别图像标注为训练图像或测试图像，以便基于待识别图像对物品检测模型和物品识别模型进行训练或测试。
在一些实施例中，基于物品检测模型确定各物品的尺寸信息，根据各物品的类别和尺寸匹配对应的属性信息；判断待识别图像中多个物品是否满足物品组合，若多个物品满足物品组合，则匹配物品组合对应的属性信息；判断待识别图像中多个物品对应的属性和，是否满足预设条件，若属性和满足预设条件，则根据预设条件对属性和进行处理；以及确定物品匹配属性信息的匹配时间，根据匹配时间确定各物品对应的属性信息。
根据本公开的另一方面，还提出一种物品识别系统，包括：图像获取模块，被配置为获取一个或多个待识别图像，其中，待识别图像中包括一个或多个待识别的物品； 预识别模块，被配置为利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值；以及物品确定模块，被配置为在概率大于阈值的情况下，识别各物品的类别。
在一些实施例中，预识别模块还被配置为若连续多个待识别图像中，第一个图像清晰并且包含完整物品的概率大于第一阈值，且其他图像清晰并且包含完整物品的概率大于第二阈值，则将连续多个图像中的第一个图像发送至物品确定模块；以及物品确定模块被配置为对连续多个图像中的第一个图像中包含的物品进行类别识别。
在一些实施例中，预识别模块还被配置为将样本图像中图像清晰并且包含完整物品的图像标注为正样本图像，将样本图像中不属于正样本图像的图像标注为负样本图像；以及基于正样本图像和负样本图像对预识别模型进行训练，以便根据训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值。
在一些实施例中，物品确定模块包括：物品检测模块，被配置为将待识别图像输入至物品检测模型，基于物品检测模型提取待识别图像中各物品对应的区域信息和第一级类别；物品管理模块，被配置为确定第一级类别中的有效类别；以及物品识别模块，被配置为将待识别图像中属于有效类别的物品的区域信息输入至物品识别模型，基于物品识别模型提取各区域信息对应的物品特征，将各区域信息对应的物品特征与物品特征库中的物品特征进行比对，确定待识别图像中各物品的第二级类别。
在一些实施例中，物品管理模块，被配置为确定第一级类别中的有效类别；物品检测模块，被配置为将待识别图像输入至物品检测模型，提取待识别图像中各物品对应的区域信息和第一级类别，调用物品管理模块，将属于有效类别的物品的区域信息输入至物品识别模块；以及物品识别模块还被配置为对样本图像中的有效类别物品的区域信息对应的物品特征进行标注生成第二标注信息，基于样本图像和第二标注信息对物品识别模型进行训练，以便根据训练好的物品识别模型提取待识别图像中各物品的区域信息对应的物品特征。
在一些实施例中，物品管理模块被配置为确定预定时间内物品特征库中的有效物品特征；以及物品识别模块还被配置为将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别。
在一些实施例中，物品识别模块被配置为确定各物品的物品特征与物品特征库中的物品特征的最小距离；若最小距离小于等于距离阈值，则将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别作为各物品的第二级类别；若最小距 离大于距离阈值，则向用户提示是否需要输入物品类别和属性信息；若需要输入物品类别和属性，则增加物品类别和属性信息，否则，将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别作为各物品的第二级类别。
在一些实施例中，属性匹配单元，被配置为根据各物品的类别匹配对应的属性信息。
在一些实施例中，物品管理模块还被配置为在匹配属性信息后，响应于用户修改物品的类别对应的属性信息，将待识别图像标注为训练图像或测试图像，以便基于待识别图像对物品检测模型和物品识别模型进行训练或测试。
在一些实施例中，属性匹配单元还被配置为以下至少一项：根据各物品的类别和尺寸匹配对应的属性信息，其中，物品检测模块还被配置为基于物品检测模型确定各物品的尺寸信息；判断待识别图像中多个物品是否满足物品组合，若多个物品满足物品组合，则匹配物品组合对应的属性信息；判断待识别图像中多个物品对应的属性和，是否满足预设条件，若满足预设条件，则根据预设条件对属性和进行处理；以及确定物品匹配属性信息的匹配时间，根据匹配时间确定各物品对应的属性信息。
根据本公开的另一方面，还提出一种物品识别系统，包括：存储器；以及耦接至存储器的处理器，处理器被配置为基于存储在存储器的指令执行上述的物品识别方法。
根据本公开的另一方面，还提出一种非瞬时性计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现上述的物品识别方法。
通过以下参照附图对本公开的示例性实施例的详细描述，本公开的其它特征及其优点将会变得清楚。
附图说明
构成说明书的一部分的附图描述了本公开的实施例，并且连同说明书一起用于解释本公开的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本公开，其中：
图1为本公开物品识别方法的一些实施例的流程示意图。
图2为本公开物品识别方法的另一些实施例的流程示意图。
图3为本公开物品识别系统的一些实施例的结构示意图。
图4为本公开物品识别系统的另一些实施例的结构示意图。
图5为本公开物品识别系统的另一些实施例的结构示意图。
图6为本公开物品识别系统的另一些实施例的结构示意图。
图7为本公开物品识别系统的另一些实施例的结构示意图。
图8为本公开物品识别系统的另一些实施例的结构示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为授权说明书的一部分。
在这里示出和讨论的所有示例中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它示例可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
为使本公开的目的、技术方案和优点更加清楚明白，以下结合具体实施例，并参照附图，对本公开进一步详细说明。
发明人发现，通过传感器触发图像采集装置对菜品进行拍照，然后进行菜品识别的方式，由于需要配置传感器，导致成本增长。并且，当识别区域有杂物时，会造成误触发。另外，在顾客在识别区域放置好托盘后，传感器从开始响应到状态稳定需要一段时间，因此，会有触发时延，影响客户体验。
而对采集到的每一张图像进行识别，对服务器的算法要求较高，并且，对所有图像进行识别也会影响识别的准确度。
图1为本公开物品识别方法的一些实施例的流程示意图。
在步骤110，获取一个或多个待识别图像，其中，待识别图像中包括一个或多个 待识别的物品。例如，在餐厅中，顾客购买了一份菜和一碗米饭，可以将菜和米饭放置到识别区域，通过摄像头拍摄识别区域，可以得到包含菜和米饭的图像。
在步骤120，利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值。
在一些实施例中，可以预先训练预识别模型，收集一定数量的样本图像，并对样本图像进行分类。例如，将样本图像中图像清晰并且包含完整物品的图像标注为正样本图像，将样本图像中不属于正样本图像的图像标注为负样本图像，基于正样本图像和负样本图像对预识别模型进行训练。将预识别模块输出结果与样本对应信息进行比较，判断比较结果是否满足构建预识别模型的损失函数的要求，反复迭代，优化和调整预识别模块的参数，使得比较结果最终满足构建预识别模型的损失函数的要求，保存该预识别模型。
例如，顾客购买了一份菜和一碗米饭，先将菜和米饭放置到托盘中，然后将托盘放置到识别区域。在用户放置托盘的过程中，托盘是不断移动的，如果图像采集装置采集图像时，托盘恰好还未完全进入到识别区域，则图像中托盘是不完整的。例如，有部分菜品未被采集到，或者，部分菜品只被采集到一小部分，这会影响后续识别的准确性。并且，托盘在移动过程中，采集到的图像带有运动模糊，这也会影响后续识别的准确性。因此，排除无效图像，只对包含完整托盘并且图像清晰的图像进行菜品识别。
在顾客正常结算过程中，采集一定数量的识别区域的图像，包括区域内无托盘的图像、托盘刚进入识别区域的图像、托盘一半进入识别区域的图像、托盘完全进入识别区域的图像。然后对图像进行分类，将图像清晰并且图像中托盘完全进入识别区域的图像标注为正样本图像，将其他图像标注为负样本图像，然后根据正负样本图像训练预识别模型。如果顾客没有用到托盘，则将图像清晰，并且包含完整菜品、饮料等计价商品的图像标注为正样本图像，其他图像标注为负样本图像，然后对预识别模型进行训练。
在步骤130，在概率大于阈值的情况下，识别各物品的类别。即该实施例中，不是对所有图像都进行物品类别识别，而是先判断图像是否符合要求，对符合要求的图像进行物品识别。
在上述实施例中，对图像清晰并且包含完整物品的概率大于阈值的图像中的各物品进行识别，而不是对所有图像都进行物品识别，能够提高识别系统的准确性以及识 别效率。
在一些实施例中，为了进一步减少图像处理负担，若连续多个待识别图像中，各图像清晰并且包含完整物品的概率都大于阈值，则对连续多个图像中的第一个图像中包含的物品进行类别识别。
在一些实施例中，为了进一步减少图像处理负担以及提高系统稳定性，若连续多个待识别图像中，第一个图像清晰并且包含完整物品的概率大于第一阈值，且其他图像清晰并且包含完整物品的概率大于第二阈值，则对连续多个图像中的第一个图像中包含的物品进行类别识别。
例如，计算当前图像清晰并且包含完整物品的概率，在判断出第一个图像清晰并且包含完整物品的概率大于0.9的图像的情况下，对该第一个图像包含的物品进行类别识别，在第二至第N张图像清晰并且包含完整物品的概率都大于0.1的情况下，不对该第二至第N张图像进行处理。
图2为本公开物品识别方法的另一些实施例的流程示意图。
在步骤210，获取一个或多个待识别图像，其中，待识别图像中包括一个或多个待识别的物品。
在步骤220，利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值。
在步骤230，在概率大于阈值的情况下，将待识别图像输入至物品检测模型，基于物品检测模型提取待识别图像中各物品对应的区域信息和第一级类别。第一级类别指物品所属的大类，例如菜品类、水果类、饮料类。
在一些实施例中，可以预先训练物品检测模型，对样本图像中的物品对应的区域信息和第一级类别进行标注，生成第一标注信息，基于样本图像和第一标注信息对物品检测模型进行训练。将物品检测模型输出结果与第一标注信息进行比较，判断比较结果是否满足构建物品识别模型的损失函数的要求，反复迭代，优化和调整物品检测模型的参数，使得比较结果最终满足构建物品检测模型的损失函数的要求，保存该物品检测模型。
在餐厅中，识别区域可能存在菜品、酸奶、水果、饮料等计价物品，也可能存在钥匙、工牌、钱包、手机、筷子、勺子、手等不计价物品。因此，可以先确定图像中各物品的大类，以便去除无效类别。
在训练物品检测模型时，将收集的图像中的物品标注为菜品、酸奶、水果、饮料、 钥匙、工牌、钱包、手机、筷子、勺子、手等类别。然后将图像输入至物品检测模型进行训练，在训练好物品检测模型后，当输入一张图像时，物品检测模型可以输出图像中各物品的区域信息和类别信息。
在步骤240，根据配置信息确定第一级类别中的有效类别。例如，对与不计价物品，有概率被被误为计价物品，因此，需要去除不计价物品的类别，仅保留计价物品的类别，避免误识。
在步骤250，将待识别图像中属于有效类别的物品的区域信息输入至物品识别模型，基于物品识别模型提取各区域信息对应的物品特征，将各区域信息对应的物品特征与物品特征库中的物品特征进行比对，确定待识别图像中各物品的第二级类别。第二级类别可以对应物品的具体信息。例如，某菜品具体为炒青椒还是炒白菜。
在一些实施例中，对样本图像中的有效类别物品的区域信息对应的物品特征进行标注，生成第二标注信息，基于样本图像和第二标注信息对物品识别模型进行训练。将物品识别模型输出结果与第二标注信息进行比较，判断比较结果是否满足构建物品识别模型的损失函数的要求，反复迭代，优化和调整物品识别模型的参数，使得比较结果最终满足构建物品识别模型的损失函数的要求，保存该物品识别模型。
例如，对待出售的菜品进行注册，先采集该菜品的图像，将该图像输入至物品检测模块，物品检测模型输出该菜品的区域信息和类别。然后，将该菜品的区域信息对应的菜品特征进行标注，并将图像和标注信息输入至物品识别模型，训练该物品识别模型。将菜品特征存入特征库，当将某菜品对应的区域信息输入至物品识别模型时，物品识别模型调用特征库，将输出的菜品特征与特征库中保存的菜品特征进行比对，识别出该菜品对应的具体信息，例如，该菜品是炒白菜还是炒青椒。
在上述实施例中，先对图像进行预识别，去除不符合要求的图像，然后识别符合要求的图像中各物品的大类。去除无效类别，仅识别属于有效类别的物品的区域信息对应的物品特征，根据物品特征能够识别出具体物品，提高物品识别的准确性。
在本公开的另一些实施例中，确定预定时间内物品特征库中的有效物品特征；将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别。
例如，物品特征库中保存了各个时期各菜品的特征，但不同季节，构成某一菜品的蔬菜可能略有差别，或者，在某些时段，某些菜品不再售卖。因此，可以将当前不参与售卖的菜品特征设置为无效特征，参与售卖的菜品特征设置为有效特征。在识别 菜品时，将待识别的菜品特征与特征库中有效菜品特征进行比对，确定该菜品具体为什么菜品。
在上述实施例中，将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别，能够减少物品识别过程中的干扰，进一步提高识别的准确率。
在本公开的一些实施例中，在将物品特征与特征库的物品特征进行比对时，先确定各物品的物品特征与物品特征库中的物品特征的最小距离；若最小距离小于等于距离阈值，则将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别；若最小距离大于距离阈值，则向用户提示是否需要输入物品类别和属性信息；若需要输入物品类别和属性，则增加物品类别和属性信息，否则，将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别。
距离例如为欧式距离，距离的大小代表相似度的大小。距离越小，说明待识别物品的物品特征与物品特征库中的物品特征越相似。在距离超过距离阈值时，说明物品特征库中可能不包含待识别物品的特征，因此，可以向用户提示是否需要输入物品类别和属性信息。若用户进行输入，则说明需要注册一个新物品，若用户没有进行输入，则将与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别。
在本公开的另一些实施例中，在识别出各物品的类别后，匹配该物品对应的属性信息。在一些实施例中，属性信息例如为价格。例如，识别出某菜品为炒白菜后，则可以匹配该菜品对应的价格，在结算时，若有多个菜品，则可以对多个菜品进行结算。
在该实施例中，由于提高了物品识别的准确性，因此，能够更加准确的匹配物品的属性信息。在属性信息为价格信息时，能够提高商品结算的准确性。
在本公开的另一些实施例中，在匹配属性信息后，响应于用户修改物品的类别对应的属性信息，将待识别图像标注为训练图像或测试图像，以便基于待识别图像对物品检测模型和物品识别模型进行训练或测试。例如，识别出某菜品为炒白菜，并且匹配出炒白菜的价格，但在实际计算时，用户修改了结算价格，则说明该菜品被识别错误。因此，可以将包含该菜品的图像作为训练图像或测试图像，并利用该图像对物品检测模型和物品识别模型进行训练或测试，通过模型的自动迭代，可以提高模型的识别的准确性。
在本公开的另一些实施例中，基于物品检测模型确定各物品的尺寸信息，根据各物品的类别和尺寸匹配对应的属性信息。例如，属性信息为价格，针对大小份的菜品，可以计算大小份菜品的尺寸边界，即大份菜品尺寸的平均值和小份菜品的平均值。将识别出的菜品的尺寸与尺寸边界进行比对，确定识别出的菜品是大份菜还是小份菜，然后匹配对应的价格。
在本公开的另一些实施例中，判断待识别图像中多个物品是否满足物品组合，若多个物品满足物品组合，则匹配物品组合对应的属性信息。例如，在餐厅结算时，配置了套餐信息，若单独点一份炒白菜15元，单独点一碗米饭2元，同时点一份炒白菜和一份米饭16元，则识别出图像中包含炒白菜和米饭后，需要匹配16元的价格。
在本公开的另一些实施例中，判断待识别图像中多个物品对应的属性和，是否满足预设条件，若属性和满足预设条件，则根据预设条件对属性和进行处理。例如，餐厅在售卖菜品时，可能存在满赠活动，例如，满20送饮料。因此，在识别出的多个菜品对应的价格之和大于20元，则可以赠送饮料。
在本公开的另一些实施例中，确定物品匹配属性信息的匹配时间，根据匹配时间确定各物品对应的属性信息。例如，在餐厅结算时，可以配置折扣时段以及折扣力度，确定菜品匹配价格的时间是否在折扣时段，若是，则可以将菜品与折扣时段对应的折扣价格进行匹配。
图3为本公开物品识别系统的一些实施例的结构示意图。该系统包括图像获取模块310、预识别模块320和物品确定模块330。
图像获取模块310被配置为获取一个或多个待识别图像，其中，待识别图像中包括一个或多个待识别的物品。
预识别模块320被配置为利用训练好的预识别模型，判断待识别图像清晰并且包含完整物品的概率是否大于阈值。
在一些实施例中，可以预先训练预识别模型，收集一定数量的样本图像，并对样本图像进行分类。例如，将样本图像中图像清晰并且包含完整物品的图像标注为正样本图像，将样本图像中不属于正样本图像的图像标注为负样本图像，基于正样本图像和负样本图像对预识别模型进行训练。
物品确定模块330被配置为在概率大于阈值的情况下，识别各物品的类别。即该实施例中，不是对所有图像都进行物品类别识别，而是先判断图像是否符合要求，对符合要求的图像进行物品识别。
在上述实施例中，对图像清晰并且包含完整物品的概率大于阈值的图像中的各物品进行识别，而不是对所有图像都进行识别，能够提高识别系统的准确性以及识别效率。
在本公开的另一些实施例中，预识别模块320还被配置为若连续多个待识别图像中，第一个图像清晰并且包含完整物品的概率大于第一阈值，且其他图像清晰并且包含完整物品的概率大于第二阈值，则将连续多个图像中的第一个图像发送至物品确定模块330。物品确定模块330被配置为对连续多个图像中的第一个图像中包含的物品进行类别识别。
例如，计算当前图像清晰并且包含完整物品的概率，在判断出第一个图像清晰并且包含完整物品的概率大于0.9的图像的情况下，对该第一个图像包含的物品进行类别识别。在第二至第N张图像清晰并且包含完整物品的概率都大于0.1的情况下，不对该第二至第N张图像进行处理，能够减少物品识别系统的处理负担以及提高系统稳定性。
图4为本公开物品识别系统的另一些实施例的结构示意图。该系统中物品确定模块330包括物品检测模块331、物品管理模块332和物品识别模块333。
物品检测模块331被配置为将待识别图像输入至物品检测模型，基于物品检测模型提取待识别图像中各物品对应的区域信息和第一级类别，调用物品管理模块332，并将属于有效类别的物品的区域信息输入至物品识别模块333。第一级类别指物品所属的大类，例如菜品类、水果类、饮料类。
在一些实施例中，可以预先训练物品检测模型，对样本图像中的物品对应的区域信息和第一级类别进行标注，生成第一标注信息，基于样本图像和第一标注信息对物品检测模型进行训练。
物品管理模块332被配置为确定第一级类别中的有效类别。在餐厅中，识别区域可能存在菜品、酸奶、水果、饮料等计价物品，也可能存在钥匙、工牌、钱包、手机、筷子、勺子、手等不计价物品。因此，在识别出物品的第一级类别后，先去除无效类别，仅保留有效类别。
物品识别模块333被配置为将待识别图像中属于有效类别的物品的区域信息输入至物品识别模型，基于物品识别模型提取各区域信息对应的物品特征，将各区域信息对应的物品特征与物品特征库中的物品特征进行比对，确定待识别图像中各物品的第二级类别。第二级类别可以对应物品的具体信息。例如，某菜品具体为炒青椒还是炒 白菜。
在一些实施例中，对样本图像中的有效类别物品的区域信息对应的物品特征进行标注，生成第二标注信息，基于样本图像和第二标注信息对物品识别模型进行训练。
在上述实施例中，先对图像进行预识别，去除不符合要求的图像，然后识别符合要求的图像中各物品的大类。去除无效类别，仅识别属于有效类别的物品的区域信息对应的物品特征，根据物品特征能够识别出具体物品，提高物品识别的准确性。
在本公开的另一些实施例中，物品管理模块332还被配置为确定预定时间内物品特征库中的有效物品特征。物品识别模块333还被配置为将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别。
例如，物品特征库中保存了各个时期各菜品的特征，但不同季节，构成某一菜品的蔬菜可能略有差别，或者，在某些时段，某些菜品不再售卖。因此，可以将当前不参与售卖的菜品特征设置为无效特征，参与售卖的菜品特征设置为有效特征。在识别菜品时，将待识别的菜品特征与特征库中有效菜品特征进行比对，确定该菜品具体为什么菜品。
在上述实施例中，将各区域信息对应的物品特征与物品特征库中的有效物品特征进行比对，确定待识别图中各物品的第二级类别，能够减少物品识别过程中的干扰，进一步提高识别的准确率。
在本公开的另一些实施例中，物品识别模块333被配置为确定各物品的物品特征与物品特征库中的物品特征的最小距离；若最小距离小于等于距离阈值，则将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别；若最小距离大于距离阈值，则向用户提示是否需要输入物品类别和属性信息；若需要输入物品类别和属性，则增加物品类别和属性信息，否则，将物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别。
距离例如为欧式距离，距离的大小代表相似度的大小。距离越小，说明待识别物品的物品特征与物品特征库中的物品特征越相似。在距离超过距离阈值时，说明物品特征库中可能不包含待识别物品的特征，因此，可以向用户提示是否需要输入物品类别和属性信息。若用户进行输入，则说明需要注册一个新物品，若用户没有进行输入，则将与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别。
在本公开的另一些实施例中，如图5所示，该系统还包括属性匹配单元510，被 配置为根据各物品的类别匹配对应的属性信息。在一些实施例中，属性信息例如为价格。例如，识别出某菜品为炒白菜后，则可以匹配该菜品对应的价格，在结算时，若有多个菜品，则可以对多个菜品进行结算。
在该实施例中，由于提高了物品识别的准确性，因此，能够更加准确的匹配物品的属性信息。在属性信息为价格信息时，能够提高商品结算的准确性。
在本公开的另一些实施例中，物品管理模块332还被配置为在匹配属性信息后，响应于用户修改物品的类别对应的属性信息，将待识别图像标注为训练图像或测试图像，以便基于待识别图像对物品检测模型和物品识别模型进行训练或测试。例如，识别出某菜品为炒白菜，并且匹配出炒白菜的价格，但在实际计算时，用户修改了结算价格，则说明该菜品被识别错误。因此，可以将包含该菜品的图像作为训练图像或测试图像，并利用该图像对物品检测模型和物品识别模型进行训练或测试，通过模型的自动迭代，可以提高模型的识别的准确性。
在本公开的另一些实施例中，属性匹配单元510还被配置为根据各物品的类别和尺寸匹配对应的属性信息，其中，物品检测模块331还被配置为基于物品检测模型确定各物品的尺寸信息。例如，属性信息为价格，针对大小份的菜品，可以计算大小份菜品的尺寸边界，即大份菜品尺寸的平均值和小份菜品的平均值。将识别出的菜品的尺寸与尺寸边界进行比对，确定识别出的菜品是大份菜还是小份菜，然后匹配对应的价格。
在本公开的另一些实施例中，属性匹配单元510还被配置为判断待识别图像中多个物品是否满足物品组合，若多个物品满足物品组合，则匹配物品组合对应的属性信息。
在本公开的另一些实施例中，属性匹配单元510还被配置为判断待识别图像中多个物品对应的属性和，是否满足预设条件，若满足预设条件，则根据预设条件对属性和进行处理。
在本公开的另一些实施例中，属性匹配单元510还被配置为确定物品匹配属性信息的匹配时间，根据匹配时间确定各物品对应的属性信息。
下面将以物品识别系统应用到餐厅结算领域为例对本公开进行介绍。
如图6所示，该实施例中包括注册模块610、预识别模块620、物品检测模块630、物品识别模块640、物品管理模块650、搜索模块660、特征库670和结算模块680。结算模块680对应属性匹配单元510。
首先，需要在系统中注册各种商品，注册模块610调用摄像头，采集结算区域的图像。为了后续识别准确，在注册菜品、饮料等计价商品时，结算区域仅放置一件商品。例如，仅放置一盘炒白菜。注册模块610将图像输入至物品检测模块630，物品检测模块630检测该商品的区域信息，并将区域信息发送至物品识别模块640，物品识别模块640提取该商品的特征，然后将特征存入特征库670中。
在顾客结账时，将商品拿到结算台，结算模块680调用摄像头拍摄图像后，摄像头将图像发送至预识别模块620判断该图像是否可用。即判断该图像清晰并且包含完整商品的概率是否大于阈值，以及该图像是否为连续多个大于阈值的图像中的第一张图像，若是，预识别模块620则将该图像发送至物品检测模块630。物品检测模块630检测该图像中包含的各物品的类别并输出各物品对应的区域信息。在拍摄商品时，无需使用传感器触发摄像头，因此，降低了成本，并且相对于设置传感器，提高了响应效率。
在菜品注册和菜品识别时，都需要对采集到的图像进行菜品检测。菜品注册台通常是放在后厨内部，方便餐厅人员注册菜品。但是后厨通常比较杂乱，可能会有一些无关的东西出现在注册台附近，如果不能过滤非菜品等非计价商品，那么就有可能将非计价商品录入到特征库中，引起误识。菜品识别时，采集到的图像除了包含菜品外，往往还包含筷子、勺子、工牌、手机、钱包、手等物品，这些非计价商品有概率被检测为菜品，引起误识。调用物品管理模块650，能够去除非计价物品，解决商品检测易受干扰的问题。
物品管理模块650可以配置哪些种类为非计价商品。例如，比如有些餐厅有饮料售卖，有些餐厅没有饮料售卖，那么餐厅可以根据实际情况配置饮料是否参与计价。再比如餐厅有送水果的活动，那么可以配置水果不参与计价。另外，物品管理模块650中可以配置钥匙、工牌、钱包、手机、筷子、勺子、手等为不计价物品。
一个餐厅在一个时段售卖的菜品可能有数十道，一年售卖的菜品种类可能成千上万，而特征库中也保存着等量的菜品特征，其中不乏一些非常相似的菜品。如果用全量特征库去实现菜品识别，容易引起误识。因此，物品管理模块650还可以将当前不参与售卖的商品的特征设置为无效特征。例如，在物品管理模块650录入每天每个时段售卖的菜品及其价格，并通过定时器触发菜单同步。同步时，首先会将特征库670中所有的商品特征置为无效，然后根据录入的菜单信息，将当前时段售卖的商品的特征置为有效，得到有效商品特征库和无效商品特征库，解决相似商品易误识的问题。 物品管理模块650还可以处理每日订单数据，统计菜品的出售情况以及顾客的订单信息等。
物品识别模块640确定计价类别商品的区域信息对应的特征信息，并通过搜索模块660调用特征库670，在特征库670找到与该商品特征最接近的特征，以便物品识别模块640输出该商品对应的具体类别，并将商品信息发送至结算模块680。
基于菜品识别的结算系统通常需要在开餐前完成菜品注册。但是在餐厅实际使用中，有些菜品，例如临时菜是在开餐后一段时间才供应，这些临时菜无法在开餐前完成注册，因此临时菜在结算时，是无法识别的。
在一些实施例中，若物品识别模块640确定该商品特征与特征库670中距离最近的特征的距离大于距离阈值，则可以向用户进行提示，例如提示结算员是否注册临时菜。若注册，则通过输入菜名和价格完成临时菜的注册，并将菜品信息发送至结算模块680，若不注册，则按当前识别结果将菜品信息发送至结算模块680。该实施例能够解决无法识别临时菜的问题。
结算模块680按照商品类别和价格进行结算。
餐厅有些菜品有不同大小的规格，大小份的价格不同，比如大份八宝粥6元，小份八宝粥3元。但是大小份的菜品除了尺寸上的差异之外，外观形态基本相似，因此，需要识别出菜品的尺寸信息，并在结算模块680中设置大小份菜品价格。餐厅在售卖菜品时，可能存在套餐优惠的活动，比如清汤拉面单价9元，牛肉片单价9元，清汤拉面和牛肉片组合套餐16元，因此，在识别出菜品后，需要判断菜品是否满足套餐设定，需要在结算模块680中设置套餐价格。有些餐厅在特定时段，例如，在傍晚，可能对某些菜品进行打折，因此，还需要在结算模块680中配置打折时段和打折力度。在某些餐厅会有满赠活动，因此，还可以在结算模块680中设置满赠信息。
在上述实施例中，由于提高了商品识别的准确性，因此，能够提高商品结算的准确性，提高用户体验以及降低商品结算的成本。
在另一些实施例中，该系统还包括IoT(物联网)平台6100、标注平台6110和算法服务器6120。在进行结算时，若用户修改了结算价格，则说明图像中的商品识别错误。物品管理模块650将识别错误的图像上传至IoT平台6100，IoT平台6100将当日的错误数据提交给标注平台6110，标注平台6100完成标注后将标注好的数据返回给算法服务器6120。算法服务器6120将标注数据随机分为训练集和测试集，并进行模型训练、模型测试，提高模型迭代效率。在注册商品之前，通过算法服务器6120 对商品识别过程中的各个模型进行训练。
在一些实施例中，注册模块610、预识别模块620和结算模块680可以设置在客户端；物品检测模块630、物品识别模块640、物品管理模块650、搜索模块660和特征库670可以设置在服务器，另外，客户端中的模块可以通过业务模块690与服务器中的模块进行通信；IoT平台6100、标注平台6110和算法服务器6120可以设置在云端。
图7为本公开物品识别系统的另一些实施例的结构示意图。该系统包括存储器710和处理器720，其中：存储器710可以是磁盘、闪存或其它任何非易失性存储介质。存储器用于存储图1、2所对应实施例中的指令。处理器720耦接至存储器710，可以作为一个或多个集成电路来实施，例如微处理器或微控制器。该处理器720用于执行存储器中存储的指令。
在一些实施例中，还可以如图8所示，该系统800包括存储器810和处理器820。处理器820通过BUS总线830耦合至存储器810。该系统800还可以通过存储接口840连接至外部存储装置850以便调用外部数据，还可以通过网络接口860连接至网络或者另外一台计算机系统(未标出)，此处不再进行详细介绍。
在该实施例中，通过存储器存储数据指令，再通过处理器处理上述指令，能够提高物品识别的准确性。
在另一些实施例中，一种计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现图1、2所对应实施例中的方法的步骤。本领域内的技术人员应明白，本公开的实施例可提供为方法、装置、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用非瞬时性存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本公开是参照根据本公开实施例的方法、设备(系统)和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多 个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
至此，已经详细描述了本公开。为了避免遮蔽本公开的构思，没有描述本领域所公知的一些细节。本领域技术人员根据上面的描述，完全可以明白如何实施这里公开的技术方案。
虽然已经通过示例对本公开的一些特定实施例进行了详细说明，但是本领域的技术人员应该理解，以上示例仅是为了进行说明，而不是为了限制本公开的范围。本领域的技术人员应该理解，可在不脱离本公开的范围和精神的情况下，对以上实施例进行修改。本公开的范围由所附权利要求来限定。

Claims (13)

  1. 一种物品识别方法，包括：
    获取一个或多个待识别图像，其中，所述待识别图像中包括一个或多个待识别的物品；
    利用训练好的预识别模型，判断所述待识别图像清晰并且包含完整物品的概率是否大于阈值；以及
    在所述概率大于阈值的情况下，识别各物品的类别。
  2. 根据权利要求1所述的物品识别方法，还包括：
    若连续多个待识别图像中，第一个图像清晰并且包含完整物品的概率大于第一阈值，且其他图像清晰并且包含完整物品的概率大于第二阈值，则对连续多个图像中的第一个图像中包含的物品进行类别识别。
  3. 根据权利要求1所述的物品识别方法，其中，训练所述预识别模型包括：
    将样本图像中图像清晰并且包含完整物品的图像标注为正样本图像，将所述样本图像中不属于正样本图像的图像标注为负样本图像；以及
    基于所述正样本图像和所述负样本图像对所述预识别模型进行训练，以便根据训练好的所述预识别模型，判断所述待识别图像清晰并且包含完整物品的概率是否大于阈值。
  4. 根据权利要求1-3任一所述的物品识别方法，其中，识别各物品的类别包括：
    将所述待识别图像输入至物品检测模型，基于所述物品检测模型提取所述待识别图像中各物品对应的区域信息和第一级类别；
    确定所述第一级类别中的有效类别；以及
    将所述待识别图像中属于有效类别的物品的区域信息输入至物品识别模型，基于所述物品识别模型提取各区域信息对应的物品特征，将所述各区域信息对应的物品特征与物品特征库中的物品特征进行比对，确定所述待识别图像中各物品的第二级类别。
  5. 根据权利要求4所述的物品识别方法，其中，训练所述物品检测模型和所述物品识别模型包括：
    对样本图像中的物品对应的区域信息和第一级类别进行标注，生成第一标注信息，基于所述样本图像和所述第一标注信息对所述物品检测模型进行训练，以便根据训练好的所述物品检测模型，确定所述待识别图像中各物品对应的区域信息和第一级类别；以及
    对所述样本图像中的有效类别物品的区域信息对应的物品特征进行标注，生成第二标注信息，基于所述样本图像和所述第二标注信息对所述物品识别模型进行训练，以便根据训练好的所述物品识别模型，提取所述待识别图像中各物品的区域信息对应的物品特征。
  6. 根据权利要求4所述的物品识别方法，其中，
    确定预定时间内所述物品特征库中的有效物品特征；以及
    将所述各区域信息对应的物品特征与所述物品特征库中的有效物品特征进行比对，确定所述待识别图中各物品的第二级类别。
  7. 根据权利要求4所述的物品识别方法，其中，
    确定各物品的物品特征与所述物品特征库中的物品特征的最小距离；
    若所述最小距离小于等于距离阈值，则将所述物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别；
    若所述最小距离大于所述距离阈值，则向用户提示是否需要输入物品类别和属性信息；以及
    若需要输入物品类别和属性，则增加物品类别和属性信息，否则，将所述物品特征库中与各物品对应的物品特征距离最近的物品特征对应的类别，作为各物品的第二级类别。
  8. 根据权利要求4所述的物品识别方法，还包括：
    根据各物品的类别，匹配对应的属性信息。
  9. 根据所述权利要求8所述的物品识别方法，还包括：
    在匹配属性信息后，响应于用户修改物品的类别对应的属性信息，将所述待识别图像标注为训练图像或测试图像，以便基于所述待识别图像对所述物品检测模型和所述物品识别模型进行训练或测试。
  10. 根据权利要求8所述的物品识别方法，还包括以下至少一个步骤：
    基于所述物品检测模型，确定各物品的尺寸信息，根据各物品的类别和尺寸匹配对应的属性信息；
    判断所述待识别图像中多个物品是否满足物品组合，若多个物品满足物品组合，则匹配所述物品组合对应的属性信息；
    判断所述待识别图像中多个物品对应的属性和，是否满足预设条件，若所述属性和满足预设条件，则根据预设条件对属性和进行处理；以及
    确定物品匹配属性信息的匹配时间，根据所述匹配时间，确定各物品对应的属性信息。
  11. 一种物品识别系统，包括：
    图像获取模块，被配置为获取一个或多个待识别图像，其中，所述待识别图像中包括一个或多个待识别的物品；
    预识别模块，被配置为利用训练好的预识别模型，判断所述待识别图像清晰并且包含完整物品的概率是否大于阈值；以及
    物品确定模块，被配置为在所述概率大于阈值的情况下，识别各物品的类别。
  12. 一种物品识别系统，包括：
    存储器；以及
    耦接至所述存储器的处理器，所述处理器被配置为基于存储在所述存储器的指令执行如权利要求1至10任一项所述的物品识别方法。
  13. 一种非瞬时性计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现权利要求1至10任一项所述的物品识别方法。
PCT/CN2020/080767 2019-04-23 2020-03-24 物品识别方法和系统 WO2020215952A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910325808.X 2019-04-23
CN201910325808.XA CN111832590B (zh) 2019-04-23 2019-04-23 物品识别方法和系统

Publications (1)

Publication Number Publication Date
WO2020215952A1 true WO2020215952A1 (zh) 2020-10-29

Family

ID=72912277

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/080767 WO2020215952A1 (zh) 2019-04-23 2020-03-24 物品识别方法和系统

Country Status (2)

Country Link
CN (1) CN111832590B (zh)
WO (1) WO2020215952A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113806508A (zh) * 2021-09-17 2021-12-17 平安普惠企业管理有限公司 基于人工智能的多轮对话方法、装置及存储介质
CN116699166A (zh) * 2023-08-08 2023-09-05 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN117056547A (zh) * 2023-10-13 2023-11-14 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117372787A (zh) * 2023-12-05 2024-01-09 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113673576A (zh) * 2021-07-26 2021-11-19 浙江大华技术股份有限公司 图像检测方法、终端及其计算机可读存储介质
CN114464271A (zh) * 2022-02-08 2022-05-10 首钢集团有限公司 一种零部件材料逆向分析方法及系统
CN118097094B (zh) * 2024-04-17 2024-09-10 南京亿猫信息技术有限公司 一种物品入侵识别方法、装置及智能购物车

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141883B1 (en) * 2015-05-11 2015-09-22 StradVision, Inc. Method, hard negative proposer, and classifier for supporting to collect hard negative images using a similarity map
CN106096932A (zh) * 2016-06-06 2016-11-09 杭州汇萃智能科技有限公司 基于餐具形状的菜品自动识别系统的计价方法
CN107122730A (zh) * 2017-04-24 2017-09-01 乐金伟 自选餐厅自动计价方法
CN107145879A (zh) * 2017-06-23 2017-09-08 依通（北京）科技有限公司 一种植物种类自动识别方法及系统
CN108256474A (zh) * 2018-01-17 2018-07-06 百度在线网络技术（北京）有限公司 用于识别菜品的方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141883B1 (en) * 2015-05-11 2015-09-22 StradVision, Inc. Method, hard negative proposer, and classifier for supporting to collect hard negative images using a similarity map
CN106096932A (zh) * 2016-06-06 2016-11-09 杭州汇萃智能科技有限公司 基于餐具形状的菜品自动识别系统的计价方法
CN107122730A (zh) * 2017-04-24 2017-09-01 乐金伟 自选餐厅自动计价方法
CN107145879A (zh) * 2017-06-23 2017-09-08 依通（北京）科技有限公司 一种植物种类自动识别方法及系统
CN108256474A (zh) * 2018-01-17 2018-07-06 百度在线网络技术（北京）有限公司 用于识别菜品的方法和装置

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113806508A (zh) * 2021-09-17 2021-12-17 平安普惠企业管理有限公司 基于人工智能的多轮对话方法、装置及存储介质
CN116699166A (zh) * 2023-08-08 2023-09-05 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN116699166B (zh) * 2023-08-08 2024-01-02 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN117056547A (zh) * 2023-10-13 2023-11-14 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117056547B (zh) * 2023-10-13 2024-01-26 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117372787A (zh) * 2023-12-05 2024-01-09 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置
CN117372787B (zh) * 2023-12-05 2024-02-20 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置

Also Published As

Publication number Publication date
CN111832590A (zh) 2020-10-27
CN111832590B (zh) 2024-03-05

Similar Documents

Publication Publication Date Title
WO2020215952A1 (zh) 物品识别方法和系统
WO2019165892A1 (zh) 自动售货方法、装置和计算机可读存储介质
RU2739542C1 (ru) Система автоматической регистрации для торговой точки
US20190244055A1 (en) Method and apparatus for checkout based on image identification technique of convolutional neural network
US10628695B2 (en) Fast item identification for checkout counter
US11354549B2 (en) Method and system for region proposal based object recognition for estimating planogram compliance
CN110555356A (zh) 自助结帐系统、方法与装置
WO2019096222A1 (zh) 一种基于身份识别和商品识别的无人售货方法和设备
CN104978585A (zh) 一种自动计价方法
WO2021179138A1 (zh) 商超货架上商品的分析方法和系统
CN109766962B (zh) 一种商品识别方法、存储介质及商品识别系统
CN111126990B (zh) 物品自动识别方法、结算方法、装置、终端及存储介质
RU2724797C1 (ru) Кассовая система и способ для идентификации блюд на подносе
JP2016024596A (ja) 情報処理装置
US11562338B2 (en) Automated point of sale systems and methods
CN111460888B (zh) 基于机器学习的物品识别方法及装置
Achakir et al. An automated AI-based solution for out-of-stock detection in retail environments
JP2015194791A (ja) Ｐｏｓ端末装置
CN112508659B (zh) 商品结算处理方法、装置及计算设备、计算机存储介质
CN113362543B (zh) 结算方法、装置、电子设备以及存储介质
CN110019983A (zh) 标签结构的扩展方法、装置及电子设备
WO2021164222A1 (zh) 确定物品分析数据
JP7339630B1 (ja) 情報処理システム
JP7328642B1 (ja) 情報処理システム
CN111126110A (zh) 一种商品信息识别方法、结算方法、装置及无人零售系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20795192

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01.03.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20795192

Country of ref document: EP

Kind code of ref document: A1