CN109858565B - Home Indoor Scene Recognition Method Based on Deep Learning Fusion of Global Features and Local Item Information - Google Patents

Abstract

The invention provides a family indoor scene recognition method based on deep learning and integrating global features and local article information. The method comprises the following steps: building a training set and a testing set of the family indoor scene picture, and sending the training set into three convolutional neural networks of Alexnet, Googlnet and VGG for training and testing respectively to obtain scene characteristics; giving corresponding weights to the three types of scene features, and taking the weighted average as a global feature; training by utilizing an SSD convolutional neural network and obtaining local features of common articles in a family indoor scene; fusing global and local article characteristics by adopting a matrix splicing mode; processing the fusion result by a clustering algorithm to generate a scene classification center vector; and judging and outputting the scene category of the picture to be detected by taking the scene classification center vector as a classification standard. By using the method, the home service robot can automatically recognize scene semantics contained in the environment, and the intelligent level of the robot is improved.

Description

A family indoor scene recognition method based on deep learning integrating global features and local item information

technical field

The invention relates to the field of scene recognition, in particular to a family indoor scene recognition method based on deep learning integrating global features and local item information.

Background technique

In the field of robotics, how the robot recognizes the current environment is an extremely important issue in the field of computer vision. The research on scene recognition of home service robots helps to obtain real-time pose information of the home scene where the robot is located. It is the home service robot to build a map of the current environment and complete the The key to follow-up work. However, the current level of intelligence of home service robots is limited, and cannot accurately and quickly judge the working environment in which they are located.

Applying the convolutional neural network model in deep learning to the work scene recognition of home service robots, it can automatically learn the feature data hidden in it from a large amount of image data, and map the features to the labels one by one, realizing the image feature recognition. Effective extraction. At the same time, the objects in the scene are used as the basic features of recognition, which is consistent with the cognitive logic of human beings to the environment. Combining the global features of the scene with the local item information and feeding it into the convolutional neural network, the robot can automatically make judgments on the current working environment by acquiring judgment experience through learning.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a family indoor scene recognition method based on deep learning that integrates global features and local item information, so as to solve the problem that the current domestic service robot has a low level of intelligence and cannot respond correctly to the working environment in a timely manner. The problem of poor scene recognition ability to realize automatic classification and recognition of family scenes by service robots.

A family indoor scene recognition method based on deep learning integrating global features and local item information, including the following steps:

Step 1: Build a training set and a test set of home indoor scene pictures, send the training set to the three convolutional neural networks of Alexnet, Googlnet, and VGG at the same time to train and generate corresponding network models, call the model to identify the training set, and output to judge each picture. The confidence of each scene to which it belongs is used as the three types of scene characteristics of the training set;

Step 2, assign specific weights to the three types of scene features obtained in step 1 to do a weighted average, and the result is used as the global feature matrix of the training data set;

Step 3, use the image annotation tool to frame the items in the training set images and mark the item labels, send the generated annotation files to the SSD convolutional neural network to train the generated item detection model, call the model to identify the training set, and output the appearance in each image. The labels of various types of items and their confidences are used as the local feature matrix of the items in the training set;

Step 4, fuse the global and local item features, and splicing the global feature matrix obtained in step 2 and the item local feature matrix obtained in step 3 horizontally to generate a comprehensive feature matrix, a row vector of the matrix corresponds to the comprehensive feature of a picture in the training set , the row vector is divided into two parts according to the number of scene categories and the number of items, the first half corresponds to the global feature of the picture, and the second half corresponds to the local object feature of the picture;

Step 5: Use the clustering algorithm to classify the training set according to the scene type, randomly select the comprehensive feature of a certain picture in each type of scene as the initial vector, and calculate each feature in the comprehensive feature and each feature in the center vector group separately. The vector similarity of the feature, according to the calculation result, update the initial vector according to certain rules, iterate to the preset number of rounds, obtain the center vector representing the comprehensive features of each scene, and form the scene classification center vector group;

Step 6: After the image to be detected is processed accordingly, a comprehensive vector is obtained, the Euclidean distance between it and each vector in the scene classification center vector group is calculated respectively, and the scene category label corresponding to the scene classification center vector with the smallest distance is taken as the output of the recognition result. .

Further, in the step 1, the three types of scene features are obtained, and the specific steps are:

Step 1-1: Divide the home indoor scene into Num_scene categories, such as bathroom, bedroom, and dining room. In order to facilitate subsequent calculations, the jth scene category is named type_j, and the network retrieves several color images of each viewing angle under each scene category. The pictures are divided into Num_train pictures in the training data set and Num_test pictures in the test data set according to a certain proportion, among which, Num_scene∈N ^* , Num_test∈N ^* , Num_train∈N ^* , i={i∈[1,Num_train] ∧i∈N ^* }, j={j∈[1,Num_scene]∧j∈N ^* };

Steps 1-2, add a set of scene category labels for the training set and test set:

List_train＝{list_train ₁ ,list_train ₂ ,...,list_train _{Num_train} }

List_test=(list_test ₁ ,list_test ₂ ,......,list_test _{Num_test} )

Num_train∈N ^* , Num_test∈N ^*

Steps 1-3, after processing the training set into the corresponding data format according to the requirements of Alexnet, Googlnet, and VGG convolutional neural networks, send the training set to Alexnet, Googlnet, and VGG convolutional neural networks at the same time, and train and generate the network model Model_Alexnet respectively. Model_Googlnet, Model_VGG;

Steps 1-4, call the generated network model to identify the training set, in which the confidences of the i-th picture judged as the j-th scene category are P _{Alexnet_i_j} , P _{Googlnet_i_j} , P _{VGG_i_j} , and the confidence levels constitute the scene confidence of the picture. Degree vector:

The matrices formed by the three types of confidence vectors are respectively used as scene feature matrices:

Num_train∈N ^* , Num_scene∈N ^* , i={i∈[1,Num_train]∧i∈N ^* },

j={j∈[1,Num_scene]∧j∈N ^* }.

Further, in the step 2, the weighted average of the three types of scene features in the step 2 specifically includes the following steps:

Step 2-1, call the network model obtained in step 1-3, detect the pictures in the test set to obtain the confidence of Num_scene scenes to which each picture belongs, take the scene category corresponding to the largest one as the discrimination result, and compare it with the real label of the picture , if the same, the identification is correct; the cumulative number of correct identifications of each convolutional neural network of Alexnet, Googlnet, and VGG is recorded as: Num_Alexnet, Num_Googlnet, Num_VGG, Num_Alexnet∈N ^* , Num_Googlnet∈N ^* , Num_VGG∈N ^* ;

Step 2-2, assign weights Weight_Alexnet, Weight_Googlnet, Weight_VGG to the scene features Matrix_Alexnet, Matrix_Googlnet, and Matrix_VGG respectively, among which,

After the weighted average, the confidence of the i-th picture in the training set being judged as the j-th scene category can be expressed as:

P _{Global_i_j} = Weight_Alexnet×P _{Alexnet_i_j} +Weight_Googlnet×P _{Googlnet_i_j} +Weight_VGG×P _{VGG_i_j}

Use the new confidence to get the global feature matrix Matrix_Global:

Further, in the step 3, the acquiring local features of common items in the home indoor scene specifically includes the following steps:

Step 3-1, select the common items in the family scene, set up the item type and maximum number Max_category, Max_num respectively; use the image annotation tool to frame the items that appear in the training set pictures, record the label, number and position of the objects in the picture, Get item tag; Max_category∈N ^* , k={k∈[1,Max_category]∧k∈N ^* }, Max_num∈N ^* ;

Step 3-2, set the maximum number of each item category Max_num _k , Max_num _k ∈ N ^* , use SSD convolutional neural network to train item labels to generate item detection model Model_SSD; k={k∈[1,Max_category]∧k∈ N ^* }, r _k = {r _k ∈[1,Max_num _k ]∧r _k ∈ N ^* };

则第i张图片被识别出k类物品的置信度向量可表示为：Step 3-3, call the model recognition training set generated in step 3-2, in which the confidence of the r _{kth k} -th item being identified in the i-th picture can be expressed as:

Then the confidence vector that the i-th image is recognized as the k-type item can be expressed as:

Each item of confidence is arranged in descending order. If the items in the picture do not reach the maximum number Max_num _k , the corresponding position will be set to zero. If the number of items exceeds the limit, only the first Max_num _k items with higher confidence will be retained;

Local item feature Matrix_object composed of item confidence vector:

Max_category∈N ^* k={k∈[1,Max_category]∧k∈N ^* }

r _k ={r _k ∈[1,Max_num _k ]∧r _k ∈N ^* }

Further, in the step 4, the step 4 fuses the global and local item features, specifically:

The global feature matrix Matrix_Global obtained in step 2 and the item local feature matrix Matrix_object obtained in step 3 are horizontally spliced to generate a comprehensive feature matrix Matrix_combination:

表示训练集中第i张图片的综合特征，行向量的前半部对应图片的全局特征，后半部对应图片的局部物体特征：The ith row vector of the synthetic feature matrix

Represents the comprehensive features of the ith picture in the training set. The first half of the row vector corresponds to the global features of the picture, and the second half corresponds to the local object features of the picture:

Further, in the step 5, the custom scene classification standard specifically includes the following steps:

和

则图片中识别出第k类物品的置信度向量可表示为：Step 5-1, divide the training data set into Num_scene parts according to the scene, the number of pictures corresponding to each scene type is Num_j, and also divide the comprehensive feature matrix according to the scene type to obtain the comprehensive feature corresponding to each scene type. In the comprehensive feature corresponding to the sub-dataset whose scene category is type_j, the confidence of the i-th _{type_j} picture being identified as the j-th scene type and the confidence of identifying the rk-th _k -th item in this picture can be respectively Expressed as

and

Then the confidence vector of identifying the k-th item in the picture can be expressed as:

The comprehensive feature matrix Matrix_combination _{type_j} corresponding to the sub-dataset whose scene category is type_j can be expressed as:

i _{type_j} ={i _{type_j∈} [1,Num_j]∧i _{type_j∈N} ^* },Max_category∈N ^* ;

Step 5-2, in each of the Num_scene sub-combination features Matrix_combination _{type_j} obtained in step 5-1, a row vector is randomly selected. In the present invention, the first comprehensive feature under each type of scene is exemplarily selected as the initial center vector, then The center vector representing the type_j scene can be expressed as:

The center vector group Matrix_center is formed by the center vector corresponding to each scene:

Num_scene∈N ^* , i _{type_j} ={i _{type_j∈} [1,Num_j]∧i _{type_j∈N} ^* },

Max_category∈N ^* ;

Step 5-3: Calculate the Euclidean distance between each feature in the comprehensive feature and each feature in the center vector group, and get Num_scene distances, and take the scene category corresponding to the minimum value as the label List_detect set obtained by scene classification :

List_detect={list_detect ₁ ,list_detect ₂ ,...,list_detect _{Num_train} };

Compare List_detect with the original scene label List_train of the picture, and update the center vector according to the following rules:

If list_detect _i =list_train _i =j, then:

If list_detect _i ≠list_train _i =j, then:

Among them, i={i∈[1,Num_train]∧i∈N ^* }, j={j∈[1,Num_scene]∧j∈N ^* }, γ is the update coefficient;

Step 5-4, set the iteration update times as Max_interation, repeat steps 5-2 and 5-3, and iterate Max_interation times to obtain the final result Matrix_center_Max_interation as the scene classification standard.

Max_interation∈N ^*

i _iteration = { i _iteration ∈ [1, Max_iteration] ∧ i _iteration ∈ N ^* }

Further, in the step 6, the test obtains the scene classification result, which specifically includes the following steps:

Step 6-1, the image to be recognized is processed through the steps of steps 1-4 in turn to obtain comprehensive features;

Step 6-2, calculate the Euclidean distance set of each vector in the Matrix_center_Max_interation obtained in step 5-4 and the comprehensive feature obtained in step 6-1:

Distance={d ₁ ,d ₂ ,...d _j ...,d _{Num_scene} }, Num_scene∈N ^*

The scene category label corresponding to the minimum value of Num_scene distances is taken as the recognition result output.

The invention considers the influence of objects in the scene on the scene type, and introduces a deep learning convolutional neural network, so that the home service robot can autonomously learn the home scene, obtain recognition experience, and realize automatic judgment of the scene. A home service robot with environmental awareness can switch the work mode to select work content according to the work environment, complete human-computer interaction and other tasks, and meet the needs of human-computer integration in the family scene. Solve the problem that the current level of intelligence of home service robots is not high, unable to respond correctly to the working environment in a timely manner, and the scene recognition ability is poor, so as to realize the automatic classification and recognition of home scenes by service robots.

Description of drawings

FIG. 1 is a system frame diagram of the present invention.

Detailed ways

The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

A family indoor scene recognition method based on deep learning integrating global features and local item information, including the following steps:

Step 1: Build a training set and a test set of home indoor scene pictures, send the training set to the three convolutional neural networks of Alexnet, Googlnet, and VGG at the same time to train and generate corresponding network models, call the model to identify the training set, and output to judge each picture. The confidence of each scene is used as the three types of scene features of the training set.

In the step 1, the three types of scene features are obtained, and the specific steps are:

Step 1-1: Divide the home indoor scene into Num_scene categories, such as bathroom, bedroom, and dining room. In order to facilitate subsequent calculations, the jth scene category is named type_j, and the network retrieves several color images of each viewing angle under each scene category. The pictures are divided into Num_train pictures in the training data set and Num_test pictures in the test data set according to a certain proportion, among which, Num_scene∈N ^* , Num_test∈N ^* , Num_train∈N ^* , i={i∈[1,Num_train] ∧i∈N ^* }, j={j∈[1,Num_scene]∧j∈N ^* }.

Steps 1-2, add a set of scene category labels for the training set and test set:

List_train＝{list_train ₁ ,list_train ₂ ,...,list_train _{Num_train} }

List_test=(list_test ₁ ,list_test ₂ ,......,list_test _{Num_test} )

Num_train∈N ^* , Num_test∈N ^*

Steps 1-3, after processing the training set into the corresponding data format according to the requirements of Alexnet, Googlnet, and VGG convolutional neural networks, send the training set to Alexnet, Googlnet, and VGG convolutional neural networks at the same time, and train and generate the network model Model_Alexnet respectively. Model_Googlnet, Model_VGG.

Steps 1-4, call the generated network model to identify the training set, in which the confidences of the i-th picture judged as the j-th scene category are P _{Alexnet_i_j} , P _{Googlnet_i_j} , P _{VGG_i_j} , and the confidence levels constitute the scene confidence of the picture. Degree vector:

The matrices formed by the three types of confidence vectors are respectively used as scene feature matrices:

Num_train∈N ^* , Num_scene∈N ^* , i={i∈[1,Num_train]∧i∈N ^* }, j={j∈[1,Num_scene]∧j∈N ^* }.

Step 2, assign specific weights to the three types of scene features obtained in step 1 to do a weighted average, and the result is used as the global feature matrix of the training data set.

In the step 2, the weighted average of the three types of scene features in the step 2 specifically includes the following steps:

Step 2-1, call the network model obtained in step 1-3, detect the pictures in the test set to obtain the confidence of Num_scene scenes to which each picture belongs, take the scene category corresponding to the largest one as the discrimination result, and compare it with the real label of the picture , if the same, the identification is correct; the cumulative number of correct identifications of each convolutional neural network of Alexnet, Googlnet and VGG is recorded as: Num_Alexnet, Num_Googlnet, Num_VGG, Num_Alexnet∈N ^* , Num_Googlnet∈N ^* , Num_VGG∈N ^* .

Step 2-2, assign weights Weight_Alexnet, Weight_Googlnet, Weight_VGG to the scene features Matrix_Alexnet, Matrix_Googlnet, and Matrix_VGG respectively, among which,

After the weighted average, the confidence of the i-th picture in the training set being judged as the j-th scene category can be expressed as:

P _{Global_i_j} = Weight_Alexnet×P _{Alexnet_i_j} +Weight_Googlnet×P _{Googlnet_i_j} +Weight_VGG×P _{VGG_i_j}

Use the new confidence to get the global feature matrix Matrix_Global:

Step 3, use the image annotation tool to frame the items in the training set images and mark the item labels, send the generated annotation files to the SSD convolutional neural network to train the generated item detection model, call the model to identify the training set, and output the appearance in each image. Each category of item labels and their confidences are used as the item local feature matrix of the training set.

In the step 3, the acquiring local features of common items in the home indoor scene specifically includes the following steps:

Step 3-1, select the common items in the family scene, set up the item type and maximum number Max_category, Max_num respectively; use the image annotation tool to frame the items that appear in the training set pictures, record the label, number and position of the objects in the picture, Get item tags. Max_category∈N ^* , k={k∈[1,Max_category]∧k∈N ^* }, Max_num∈N ^* ;

Step 3-2, set the maximum number of each item category Max_num _k , Max_num _k ∈ N ^* , use SSD convolutional neural network to train item labels to generate item detection model Model_SSD. k={k∈[1,Max_category]∧k∈N ^* }, r _k ={r _k ∈[1,Max_num _k ]∧r _k ∈N ^* }

则第i张图片被识别出k类物品的置信度向量可表示为：Step 3-3, call the model recognition training set generated in step 3-2, in which the confidence of the r _{kth k} -th item being identified in the i-th picture can be expressed as:

Then the confidence vector that the i-th image is recognized as the k-type item can be expressed as:

Each item of confidence is arranged in descending order. If the items in the picture do not reach the maximum number Max_num _k , the corresponding position will be set to zero. If the number of items exceeds the limit, only the first Max_num _k items with higher confidence will be retained.

Local item feature Matrix_object composed of item confidence vector:

Max_category∈N ^* k={k∈[1,Max_category]∧k∈N ^* }

r _k ={r _k ∈[1,Max_num _k ]∧r _k ∈N ^* }

Step 4, fuse the global and local item features, and splicing the global feature matrix obtained in step 2 and the item local feature matrix obtained in step 3 horizontally to generate a comprehensive feature matrix, a row vector of the matrix corresponds to the comprehensive feature of a picture in the training set , the row vector is divided into two parts according to the number of scene categories and the number of items. The first half corresponds to the global feature of the picture, and the second half corresponds to the local object feature of the picture.

In the step 4, the step 4 fuses the global and local item features, specifically:

The global feature matrix Matrix_Global obtained in step 2 and the item local feature matrix Matrix_object obtained in step 3 are horizontally spliced to generate a comprehensive feature matrix Matrix_combination:

表示训练集中第i张图片的综合特征，行向量的前半部对应图片的全局特征，后半部对应图片的局部物体特征：The ith row vector of the synthetic feature matrix

Represents the comprehensive features of the ith picture in the training set. The first half of the row vector corresponds to the global features of the picture, and the second half corresponds to the local object features of the picture:

i={i∈[1,Num_train]∧i∈N ^* }, Max_category∈N ^*

Step 5: Use the clustering algorithm to classify the training set according to the scene type, randomly select the comprehensive feature of a certain picture in each type of scene as the initial vector, and calculate each feature in the comprehensive feature and each feature in the center vector group separately. The vector similarity of the feature, according to the calculation result, update the initial vector according to certain rules, iterate to the preset number of rounds, obtain the center vector representing the comprehensive features of each scene, and form the scene classification center vector group.

In the step 5, the custom scene classification standard specifically includes the following steps:

和

则图片中识别出第k类物品的置信度向量可表示为：Step 5-1, divide the training data set into Num_scene parts according to the scene, the number of pictures corresponding to each scene type is Num_j, and also divide the comprehensive feature matrix according to the scene type to obtain the comprehensive feature corresponding to each scene type. In the comprehensive feature corresponding to the sub-dataset whose scene category is type_j, the confidence of the i-th _{type_j} picture being identified as the j-th scene type and the confidence of identifying the rk-th _k -th item in this picture can be respectively Expressed as

and

Then the confidence vector of identifying the k-th item in the picture can be expressed as:

The comprehensive feature matrix Matrix_combination _{type_j} corresponding to the sub-dataset whose scene category is type_j can be expressed as:

i _{type_j} = {i _{type_j} ∈ [1, Num_j] ∧ i _{type_j} ∈ N ^* }, Max_category ∈ N ^*

中各随机取一个行向量，本发明中示例性地选取每一类场景下第一个综合特征作为初始中心向量，则代表type_j场景的中心向量可表示为：Step 5-2, the Num_scen e sub-synthetic features Matri_x obtained in step 5-1

A row vector is randomly selected in the present invention, and the first comprehensive feature under each type of scene is exemplarily selected as the initial center vector, then the center vector representing the type_j scene can be expressed as:

The center vector group Matrix_center is formed by the center vector corresponding to each scene:

Num_scene∈N ^* , i _{type_j} ={i _{type_j∈} [1,Num_j]∧i _{type_j∈N} ^* },

Max_category∈N ^*

Step 5-3: Calculate the Euclidean distance between each feature in the comprehensive feature and each feature in the center vector group, and get Num_scene distances, and take the scene category corresponding to the minimum value as the label List_detect set obtained by scene classification :

List_detect={list_detect ₁ ,list_detect ₂ ,......,list_detect _{Num_train} }

Compare List_detect with the original scene label List_train of the picture, and update the center vector according to the following rules:

If list_detect _i =list_train _i =j, then:

If list_detect _i ≠list_train _i =j, then:

Among them, i={i∈[1,Num_train]∧i∈N ^* }, j={j∈[1,Num_scene]∧j∈N ^* }, and γ is the update coefficient.

Step 5-4, set the iteration update times as Max_interation, repeat steps 5-2 and 5-3, and iterate Max_interation times to obtain the final result Matrix_center_Max_interation as the scene classification standard.

Max_interation∈N ^*

i _iteration = { i _iteration ∈ [1, Max_iteration] ∧ i _iteration ∈ N ^* }

Step 6: After the image to be detected is processed accordingly, a comprehensive vector is obtained, the Euclidean distance between it and each vector in the scene classification center vector group is calculated respectively, and the scene category label corresponding to the scene classification center vector with the smallest distance is taken as the output of the recognition result. .

In the step 6, the test obtains the scene classification result, which specifically includes the following steps:

In step 6-1, the pictures to be recognized are processed through the steps of steps 1-4 in sequence to obtain comprehensive features.

Step 6-2, calculate the Euclidean distance set of each vector in the Matrix_center_Max_interation obtained in step 5-4 and the comprehensive feature obtained in step 6-1:

Distance={d ₁ ,d ₂ ,...d _j ...,d _{Num_scene} }, Num_scene∈N ^*

The scene category label corresponding to the minimum value of Num_scene distances is taken as the recognition result output.

The above descriptions are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, but any equivalent modifications or changes made by those of ordinary skill in the art based on the contents disclosed in the present invention should be included in the within the scope of protection described in the claims.

Claims (4)

1. The method for identifying the family indoor scene based on the deep learning and integrating the global features and the local article information is characterized in that: the method comprises the following steps:

step 1, constructing a training set and a test set of pictures of a family indoor scene, simultaneously sending the training set into three convolutional neural networks of Alexnet, Googlnet and VGG for respective training to generate corresponding network models, calling the model to identify the training set, and outputting and judging confidence coefficients of each scene to which each picture belongs to as three types of scene features of the training set;

step 2, weighting and averaging the three types of scene features obtained in the step 1, and taking the result as a global feature matrix of a training data set;

step 3, framing the articles in the pictures of the training set by using a picture labeling tool, labeling article labels, sending the generated labeling file into an SSD convolutional neural network for training to generate an article detection model, calling the model to identify the training set, and outputting the article labels of various types appearing in each picture and the confidence thereof as an article local feature matrix of the training set;

step 4, fusing global and local article characteristics, horizontally splicing the global characteristic matrix obtained in the step two and the article local characteristic matrix obtained in the step three to generate a comprehensive characteristic matrix, wherein one row vector of the matrix corresponds to the comprehensive characteristic of one picture in the training set, the row vector is divided into two parts according to the scene category number and the article number, the first half corresponds to the global characteristic of the picture, and the second half corresponds to the local object characteristic of the picture;

In the step 4, the step 4 fuses global and local article features, specifically:

horizontally splicing the Global feature Matrix _ Global obtained in the step 2 and the article local feature Matrix _ object obtained in the step 3 to generate a comprehensive feature Matrix _ combination:

ith row vector of synthetic feature matrix

The comprehensive characteristics of the ith picture in the training set are represented, the first half of the line vector corresponds to the global characteristics of the picture, and the second half corresponds to the local object characteristics of the picture:

i＝{i∈[1,Num_train]∧i∈N ^* }，Max_category∈N ^*

step 5, classifying the training set according to scene types by using a clustering algorithm, randomly taking the comprehensive features of a certain picture under each type of scene as initial vectors, respectively calculating the vector similarity between each feature in the comprehensive features and each feature in the central vector group, updating the initial vectors according to the calculation results, iterating to preset turns to obtain central vectors representing the comprehensive features of each scene, and forming a scene classification central vector group;

in the step 5, the user-defined scene classification standard specifically includes the following steps:

step 5-1, dividing the training data set into Num _ scene parts according to scenes, wherein the number of pictures corresponding to each scene type is Num _ j, and dividing the comprehensive characteristic matrix according to the scene types to obtain comprehensive characteristics corresponding to each scene type; in the comprehensive characteristics corresponding to the subdata set with the scene type of type _ j, the ith _{type_j} Confidence of j scene type identified in picture and r-th scene type identified in picture _k The confidence of each k-type article can be respectively expressed as

And

the confidence vector identifying the kth class of item in the picture can be expressed as:

sorting out a comprehensive characteristic Matrix _ combination corresponding to a sub data set with a scene type of type _ j _{type_j} Can be expressed as:

i _{type_j} ＝{i _{type_j} ∈[1，Num_j]∧i _{type_j} ∈N ^* }，Max_category∈N ^* ；

step 5-2, the Num _ scene sub-synthesis feature Matrix _ combination obtained in the step 5-1 _{type_j} In the method, a row vector is randomly selected, the first comprehensive feature under each type of scene is selected as an initial center vector, and then the center vector representing the type _ j scene can be represented as follows:

and forming a central vector group Matrix _ center by the central vectors corresponding to the scenes:

Num_scene∈N ^* ，i _{type_j} ＝{i _{type_j} ∈[1，Num_j]∧i _{type_j} ∈N ^* }，

Max_category∈N ^* ；

step 5-3, respectively calculating the Euclidean distance between each feature in the comprehensive features and each feature in the central vector group to obtain Num _ scene distances, and taking the scene category corresponding to the minimum value as a label List _ detect set obtained by scene classification:

List_detect＝{list_detect ₁ ,list_detect ₂ ,......,list_detect _{Num_train} }；

comparing the List _ detect with the original scene tag List _ train of the picture, and updating the central vector according to the following rules:

if list _ detect _i ＝list_train _i J, then:

if list _ detect _i ≠list_train _i J, then:

wherein i ═ { i ∈ [1, Num _ train [ ] ]∧i∈N ^* }，j＝{j∈[1,Num_scene]∧j∈N ^* Gamma is an updating coefficient;

step 5-4, setting the iteration updating times as Max _ interaction, repeating the steps 5-2 and 5-3, and iterating the Max _ interaction times to obtain a final result Matrix _ center _ Max _ interaction as a scene classification standard;

Max_interation∈N ^*

i _iteration ＝{i _iteration ∈[1，Max_iteration]∧i _iteration ∈N ^* step 6, after the picture to be detected is correspondingly processed, obtaining a comprehensive vector, respectively calculating Euclidean distances between the comprehensive vector and each vector in the scene classification center vector group, and outputting a scene classification label corresponding to the scene classification center vector with the minimum distance as an identification result;

in the step 6, the test obtains a scene classification result, and specifically includes the following steps:

step 6-1, processing the picture to be identified in the steps 1-4 in sequence to obtain comprehensive characteristics;

step 6-2, respectively calculating Euclidean distance sets of the comprehensive characteristics obtained in the step 6-1 and each vector in Matrix _ center _ Max _ interaction obtained in the step 5-4:

Distance＝{d ₁ ,d ₂ ,...d _j ...,d _{Num_scene} }，Num_scene∈N ^*

and taking the scene category label corresponding to the minimum value of the Num _ scene distances as a recognition result to output.

2. The deep learning based home indoor scene recognition method fusing global features and local item information according to claim 1, characterized in that: in the step 1, three types of scene characteristics are obtained, and the specific steps are as follows:

Step 1-1, dividing the family indoor scenes into total Num _ scene categories, naming the jth scene category as type _ j for facilitating subsequent calculation, and dividing a plurality of color pictures of each view angle under each scene category, which are searched by a network, into a training data set of total Num _ train pictures and a test data set of total Num _ test pictures, wherein Num _ scene belongs to N ^* ，Num_test∈N ^* ，Num_train∈N ^* ，i＝{i∈[1,Num_train]∧i∈N ^* }，j＝{j∈[1,Num_scene]∧j∈N ^* }；

Step 1-2, adding a scene category label set for the training set and the test set:

List_train＝{list_train ₁ ,list_train ₂ ,......,list_train _{Num_train} }

List_test＝(list_test ₁ ,list_test ₂ ,......,list_test _{Num_test} )

Num_train∈N ^* ，Num_test∈N ^*

step 1-3, processing the training set into a corresponding data format according to the requirements of Alexnet, Googlnet and VGG convolutional neural networks, simultaneously sending the training set into the Alexnet, Googlnet and VGG convolutional neural networks, and respectively training and generating network models of Model _ Alexnet, Model _ Googlnet and Model _ VGG;

step 1-4, calling the generated network model identificationTraining set, wherein the confidence of the ith picture being judged as the jth scene category is P _{Alexnet_i_j} ，P _{Googlnet_i_j} ，P _{VGG_i_j} And forming a scene confidence coefficient vector of the graph by using the confidence coefficients:

the matrixes formed by the three types of confidence coefficient vectors are respectively used as scene characteristic matrixes:

Num_train∈N ^* ，Num_scene∈N ^* ，i＝{i∈[1,Num_train]∧i∈N ^* }，

j＝{j∈[1,Num_scene]∧j∈N ^* }。

3. the deep learning based home indoor scene recognition method fusing global features and local item information according to claim 1, characterized in that: in the step 2, the step 2 of weighted averaging the three types of scene features specifically includes the following steps:

Step 2-1, calling the network model obtained in the step 1-3, detecting the pictures of the test set to obtain the confidence coefficient of each Num _ scene to which each picture belongs, taking the scene type corresponding to the largest picture as a judgment result, comparing the judgment result with the picture real label, and if the judgment result is the same as the picture real label, judging that the identification is correct; the number of correct identifications of each convolutional neural network accumulated by Alexnet, Googlnet, VGG is recorded as: num _ Alexnet, Num _ Googlnet, Num _ VGG, Num _ Alexnet belonging to N ^* ，Num_Googlnet∈N ^* ，Num_VGG∈N ^* ；

Step 2-2, respectively assigning weights Weight _ alexne, Weight _ google, and Weight _ VGG to the scene features Matrix _ alexne, Matrix _ google, and Matrix _ VGG, wherein,

the confidence level that the ith picture in the training set is judged as the jth scene class after the weighted average can be expressed as:

P _{Global_i_j} ＝Weight_Alexnet×P _{Alexnet_i_j} +Weight_Googlnet×P _{Googlnet_i_j} +Weight_VGG×P _{VGG_i_j}

and obtaining a Global feature Matrix _ Global by using the new confidence coefficient:

i＝{i∈[1,Num_train]∧i∈N ^* }j＝{j∈[1,Num_scene]∧j∈N ^* }。

4. the deep learning-based home indoor scene recognition method integrating global features and local article information according to claim 1, wherein: in the step 3, local features of common articles in a family indoor scene are obtained, and the method specifically comprises the following steps:

step 3-1, selecting common articles in a family scene, and respectively setting maximum values of article types and numbers, namely Max _ category and Max _ num; using the picture marking tool frame to frame the objects appearing in the pictures of the training set, recording the labels, the number and the positions of the objects in the pictures, and obtaining object marks; max _ category belongs to N ^* ，k＝{k∈[1,Max_category]∧k∈N ^* }，Max_num∈N ^* ；

Step 3-2, setting the maximum number of each article type Max _ num _k ，Max_num _k ∈N ^* Training the article label by utilizing an SSD convolutional neural network to generate an article detection Model _ SSD; k ∈ { k ∈ [1, Max _ category [ ]]∧k∈N ^* }，r _k ＝{r _k ∈[1,Max_num _k ]∧r _k ∈N ^* }；

Step 3-3, calling the model recognition training set generated in the step 3-2, wherein the ith picture is recognized as the r < th > picture _k The individual class k item confidence may be expressed as:

the confidence vector that the ith picture is recognized as a k-type item can be expressed as:

wherein each item of confidence coefficient is arranged from large to small, if the articles in the picture do not reach the maximum number Max _ num _k If the number of articles is over-limit, only the first Max _ num with high confidence coefficient is reserved _k An item;

local item feature Matrix _ object composed of item confidence vectors:

i＝{i∈[1,Num_train]∧i∈N ^* }i＝{i∈[1,Num_train]∧i∈N ^* }

Max_category∈N ^* k＝{k∈[1,Max_category]∧k∈N ^* }

r _k ＝{r _k ∈[1,Max_num _k ]∧r _k ∈N ^* }。

Images