CN116704202A - Visual relation detection method based on knowledge embedding - Google Patents

Abstract

The invention discloses a visual relation detection method based on knowledge embedding, which comprises the following steps: the method comprises the steps of inputting images, and respectively detecting target initial category prediction features, spatial features of target boundary boxes and visual features of target pair joint areas; defining types of priori knowledge, and constructing a corresponding knowledge graph aiming at each type of priori knowledge; constructing target initial category prediction features into graph structures represented by nodes and edges, and representing knowledge graphs into adjacent matrixes corresponding to the graph structures; obtaining the mutually related target category characteristics based on an updating mechanism of the gate control graph neural network GGNN; according to the spatial characteristics of the target bounding box, the target category characteristics and the visual characteristics of the target pair joint region, the context information is obtained, and the visual relation detection is carried out on each target pair through the softmax function, so that the problems of incomplete image information capturing and understanding and low performance under complex scenes during the visual relation detection are solved.

Description

A Visual Relationship Detection Method Based on Knowledge Embedding

technical field

The invention belongs to the technical field of visual relationship detection, and in particular relates to a visual relationship detection method based on knowledge embedding.

Background technique

After nearly 70 years of rapid development, artificial intelligence has had a profound impact on mankind in terms of technology, economy, and culture. Computer vision is an important research field in artificial intelligence technology, which allows computers to simulate the capabilities of the human visual system through image or video data. At present, basic computer vision tasks such as image classification, object detection and semantic segmentation have achieved remarkable results, even surpassing human performance. However, for more complex image understanding tasks, further research results are still needed. Therefore, visual relationship detection is needed as a bridge to detect the relationship between image objects on the basis of recognizing the visual content in the image, so as to deeply understand and grasp the rich semantic information in the image.

Visual relationship detection methods are a class of computer vision techniques used to identify relationships between objects in images. The earliest visual relationship detection method refers to the superior model of the neural network in the CV field and the NLP field. Through a certain combination, a comprehensive model suitable for the joint understanding of images and semantics is constructed to complete the visual relationship detection.

These methods consist of three parts, object detection module, object feature extraction module and relational modeling module. This can be achieved by different approaches, early approaches based on translation embedding networks were used to explore visual relationship detection. In this approach, predicates in language translation are represented by vector transformations of embedded features connecting subject and object bounding box regions, thereby representing relations between entities. However, such methods are often too simplistic and independently utilize the visual features of objects extracted from images, ignoring the important role of contextual information in visual detection.

As a result, a message passing model based on recurrent neural networks is introduced to exchange relational information between nodes and edges. Subsequently, due to the large number of "subject-relation" or "relation-object" substructures in relation triplets, a global context information extraction model based on recurrent neural networks was proposed. This kind of model generally uses the model based on long short-term memory artificial neural network LSTM to extract image context information, but ignores the inherent law between image objects and relational connections, making it difficult to accurately capture key information.

As mentioned above, visual relationship detection has undergone rapid development, yielding remarkable results. Based on the continuous progress of deep learning, the target detection algorithm represented by Faster Regional Convolutional Neural Network Faster-RCNN has achieved superior performance and is widely used in actual industry. However, due to the complexity of relationships in real-world scenes and the diversity of relationships in image scenes, accurate understanding of the relationships between objects in visual images still faces great challenges. In data relation detection tasks with semantic information, methods that only rely on image features may not achieve satisfactory results. Existing technologies only use image features or directly fuse semantic features for feature enhancement, resulting in a limited number of relationships that can be detected and ambiguities in complex relationship detection. Therefore, it is particularly important to use knowledge embedding to optimize the intelligent understanding and reasoning capabilities of the model.

Contents of the invention

The invention provides a visual relationship detection method based on knowledge embedding, which solves the problems of incomplete capture and understanding of image information and low performance in complex scenes during visual relationship detection.

In order to solve the above technical problems, the technical solution of the present invention is: a visual relationship detection method based on knowledge embedding, comprising the following steps:

S1. Input the image, respectively detect the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the target pair joint area;

S2. Define the type of prior knowledge, and construct a corresponding knowledge graph for each prior type;

S3. Construct the target initial category prediction feature into a graph structure represented by nodes and edges, and represent the knowledge graph as an adjacency matrix corresponding to the graph structure, and represent edge information through the adjacency matrix;

S4. Based on the update mechanism of the gated graph neural network GGNN, the nodes of the graph structure and the adjacency matrix are jointly learned with the initial category prediction features of the target to obtain interrelated target category features;

S5. According to the spatial features of the target bounding box, the interrelated target category features and the visual features of the target pair joint area, the context information is obtained, and the relationship between each target pair is classified through the softmax function to complete the visual relationship detection.

The beneficial effects of the present invention are: the present invention constructs different types of knowledge graphs by defining the types of prior knowledge, and jointly learns vision and knowledge, and further learns the internal dependencies between various information, which solves the problem of visual relationship detection. Incomplete capture and understanding of image information and poor performance in complex scenes.

Further, the specific steps of the step S1 are:

S11, input the image, use the feature pyramid network FPN based on the deep residual network ResNet101 as the backbone network of the regional convolutional neural network Faster R-CNN, and extract the visual features of the multi-scale fusion of the image;

S12. Pass the visual features through the candidate area network to obtain a target set of interest, and use non-maximum value suppression NMS to obtain candidate areas containing target information;

S13. According to the regional feature aggregation method ROIAlign, the features of the candidate regions are uniformly represented, and the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the joint area of the target pair are respectively obtained according to the full connection and the softmax function.

Further, the specific steps of step S2 are:

S21. Define the type of prior knowledge through the source of knowledge, wherein the knowledge extracted from the Visual Genome VG dataset is defined as statistical knowledge, and the knowledge extracted from the global vector representation Glove word vector is defined as language knowledge;

S22. According to the statistical knowledge, make statistics on the co-occurrence relationship between different categories of objects in the statistical knowledge, obtain the co-occurrence probability distribution table according to the co-occurrence relationship, and measure the cosine similarity of the co-occurrence probability distribution table to obtain internal statistics knowledge map;

S23. According to the language knowledge, the language prior knowledge table is obtained through text feature embedding, and the language prior knowledge table is used to measure the cosine similarity to construct an external language knowledge graph;

S24. Construct the knowledge graph of the internal statistical knowledge graph and the external language knowledge graph through weighted fusion.

The beneficial effect of the above further solution is: the present invention builds a knowledge map through internal statistical knowledge and external language knowledge, which solves the limitations of the traditional relationship detection method in terms of generalization ability and external language ambiguity, thereby overcoming the incomplete capture and understanding of image information The problem.

Further, the cosine similarity measure formula in the step S22 is:

Among them, cos(θ _{i, j} ) represents the cosine similarity measure, δ _i represents the co-occurrence relationship of target i, δ _j represents the co-occurrence relationship of target j, θ _{i, j} represents the similarity between target i and target j, n Indicates the total number of targets.

Further, the specific steps of step S4 are:

S41. Determine the number of node feature propagation times T in step S3, control the spread range and depth of node features, and complete the definition of GGNN parameters for the gated graph neural network;

S42. Calculate the initial hidden state of the node, and jointly learn the nodes in the graph structure and the corresponding adjacency matrix in each time step t;

S43. Using the joint learning information at time step t and the node hidden state at the previous time step, according to the node information propagation mechanism of the gated graph neural network GGNN, update the hidden state of the node, and obtain the final hidden state of each node;

S44. Fusing the final hidden state of each node and the initial target category prediction features to obtain the interrelated target category features after joint learning of vision and knowledge.

The beneficial effect of the above further scheme is: the present invention jointly learns the nodes in the graph structure and the corresponding adjacency matrix, can more comprehensively capture the semantic meaning in the image, and enhance the confidence of the model in predicting the target category, thereby improving the visual The accuracy of the relationship detection method.

Further, the expression of the initial hidden state of the node in the step S42 is:

in, Represents the hidden state of the mth node at the initial moment, σ() represents the corresponding activation function, FC() represents the fully connected layer, /> Indicates that the spliced features are mapped to low-dimensional vectors, x _i represents the visual feature vector obtained by feature extraction of the i-th target, p _i represents the target initial category prediction feature of the i-th target, || represents the splicing between features operate;

The expression for joint learning in the step S42 is:

in, Represents the joint learning information of the mth node at the tth time step, N represents the number of categories, /> Indicates the hidden state of the Nth node at time step t-1, G _s indicates the constructed knowledge graph, (G _s ) _m indicates the mth column information of the knowledge graph, and b indicates hyperparameters;

The hidden state calculation process of updating nodes in the step S43 is:

in, Indicates that the mth node controls the forgotten information, σ() represents the corresponding activation function, h ^(t-1) represents the hidden state of the node at t-1 time step, h _m ^(t-1) represents the mth node at t Hidden state for -1 time step, /> Indicates the new information generated by the mth node control, /> Represents the information newly generated by the mth node, tanh() represents the tanh activation function, W ^zT represents the update gate training parameter matrix, U ^zT represents the update gate bias parameter matrix, W ^T represents the reset gate training parameter matrix, U ^T represents Reset the gate bias parameter matrix, W ^rT represents the training parameter matrix representing the new information, U ^rT represents the bias parameter matrix of the new information, ⊙ represents the product of the corresponding elements of the vector, /> Indicates the hidden state of the mth node;

The expression of the target category feature in the step S44 is:

Among them, h _dk represents the initial state of the node in area k, FC() represents the fully connected layer, Indicates the initial node hidden state, /> represents the learned final node state, /> represents rescaling, φ ₀ () represents the aggregation method, p _f represents the interrelated target category features, and p _i represents the target initial category prediction features of the i-th target.

Further, the specific steps of step S5 are:

S51. Use the attention mechanism Transformer to fuse the spatial features of the target bounding box, the visual features of the joint area of the target pair, and the interrelated target category features to obtain the context information of the target pair;

S52. Fusing the visual features of the target pair joint area with the context information of the target pair to obtain relational context features;

S53. According to the relationship context feature, use the softmax function to classify the relationship of each target pair, and complete the visual relationship detection target pair to perform relationship detection.

Further, the calculation formula of the context information of the target pair in the step S51 is:

Q＝Transformer([X, E ^p , E ^f ]; W ^z )

Among them, Q represents the context information of the target pair, that is, the fused features, W ^z represents the training parameters, E ^p represents the embedding vector of target classification prediction, E ^f represents the embedding vector of the target bounding box coordinates, X represents the visual features of the target, Transformer() represents a deep learning model that utilizes an attention mechanism.

Further, in the step S53, the calculation formula for using the softmax function to classify the relationship between each target pair is:

r _ij =q _ij ⊙e _i ⊙u _ij

p(r _k |l,o _i ,o _j )＝softmax(r _ij )

Among them, e _i represents the semantic feature of the target pair, W _z represents the training parameters, r _ij represents the context feature corresponding to target i and target j, and /> The target category features of target i and target j respectively, q _ij represents the context information of the target pair, /> Indicates rescaling, u _ij indicates the visual features of the target pair joint region, ⊙ indicates the product of corresponding elements of the vector, p(r _k |l,o _i ,o _j ) indicates the predicted probability of the target pair relationship classification, r _k indicates the target pair l represents the target image, o _i represents the target i in the target pair, and o _j represents the target j in the target pair.

The beneficial effect of the above further solution is: the present invention uses the attention mechanism Transformer to further learn the internal dependencies among various information, better capture the correlation between features, and improve the performance of the visual relationship detection method in complex scenes.

Description of drawings

Fig. 1 is a flowchart of visual relationship detection based on knowledge embedding in the present invention.

Figure 2 is a schematic representation of the graphic structure representation of the present invention.

Detailed ways

Those skilled in the art will appreciate that the embodiments described here are to help readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical revelations disclosed in the present invention without departing from the essence of the present invention, and these modifications and combinations are still within the protection scope of the present invention.

Example

As shown in Figure 1, the present invention provides a visual relationship detection method based on knowledge embedding, including the following steps:

S1. Input the image, respectively detect the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the target pair joint area;

S2. Define the type of prior knowledge, and construct a corresponding knowledge graph for each prior type;

S3. Construct the target initial category prediction feature into a graph structure represented by nodes and edges, and represent the knowledge graph as an adjacency matrix corresponding to the graph structure, and represent edge information through the adjacency matrix;

S4. Based on the update mechanism of the gated graph neural network GGNN, the nodes of the graph structure and the adjacency matrix are jointly learned with the initial category prediction features of the target to obtain interrelated target category features;

S5. According to the spatial features of the target bounding box, the interrelated target category features and the visual features of the target pair joint area, the context information is obtained, and the relationship between each target pair is classified through the softmax function to complete the visual relationship detection.

The specific steps of the step S1 are:

S11, input the image, use the feature pyramid network FPN based on the deep residual network ResNet101 as the backbone network of the regional convolutional neural network Faster R-CNN, and extract the visual features of the multi-scale fusion of the image;

S12. Pass the visual features through the candidate area network to obtain a target set of interest, and use non-maximum value suppression NMS to obtain candidate areas containing target information;

S13. According to the regional feature aggregation method ROIAlign, the features of the candidate regions are uniformly represented, and the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the joint area of the target pair are respectively obtained according to the full connection and the softmax function.

In this embodiment, the visual feature extraction is first performed on the input image, and the initial target category prediction features required for subsequent detection, the spatial features of the target bounding box, and the visual features of the joint area of the target pair are extracted. The feature pyramid network FPN based on the deep residual network ResNet101 is used as the backbone network of Faster R-CNN to obtain multi-scale fusion visual features. Then, with the help of the candidate area network, the target candidate set B={b ₁ ,b ₂ ,...,b _n } is obtained, and repeated targets are removed through sampling and non-maximum suppression NMS, and a series of inclusions for subsequent visual relationship detection are obtained. Candidate regions for target information.

Then, ROIAlign is used to divide the candidate region into k×k units, and the features of the candidate region are uniformly represented. Finally, the required visual features can be obtained by using the full connection and softmax function, which include the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the joint region of the target pair.

The specific steps of the step S2 are:

S21. Define the type of prior knowledge through the source of knowledge, wherein the knowledge extracted from the Visual Genome VG dataset is defined as statistical knowledge, and the knowledge extracted from the global vector representation Glove word vector is defined as language knowledge;

S22. According to the statistical knowledge, make statistics on the co-occurrence relationship between different categories of objects in the statistical knowledge, obtain the co-occurrence probability distribution table according to the co-occurrence relationship, and measure the cosine similarity of the co-occurrence probability distribution table to obtain internal statistics knowledge map;

S23. According to the language knowledge, the language prior knowledge table is obtained through text feature embedding, and the language prior knowledge table is used to measure the cosine similarity to construct an external language knowledge graph;

S24. Construct the knowledge graph of the internal statistical knowledge graph and the external language knowledge graph through weighted fusion.

In this embodiment, the knowledge type is first defined, and the knowledge is divided according to different sources of knowledge, which are defined as statistical knowledge and language knowledge. Among them, the knowledge extracted from the Visual Genome VG dataset is defined as statistical knowledge, and the global vector representation The knowledge extracted by Glove word vector is defined as language knowledge. Then obtain the internal statistical knowledge map, assuming that there are M categories to be detected, and count the co-occurrence relationship between each target and other targets respectively, so as to obtain a co-occurrence probability distribution table T _s of M×M size. Afterwards, cosine similarity measurement is performed on the obtained co-occurrence probability distribution table T _s to evaluate the similarity and correlation between target pairs, and obtain the statistical knowledge graph G _s .

Similarly, according to the language knowledge, the language prior knowledge table T _l is obtained through text feature embedding. The dimension used by the word vector is 300 dimensions, and the semantic relationship between the targets is captured by the cosine distance measurement to obtain the language knowledge graph G _l , and finally the weighted fusion statistical knowledge graph and the language knowledge graph are used to obtain the final knowledge graph G _a .

In the step S22, the cosine similarity measure formula is:

Among them, cos(θ _{i, j} ) represents the cosine similarity measure, δ _i represents the co-occurrence relationship of target i, δ _j represents the co-occurrence relationship of target j, θ _{i, j} represents the similarity between target i and target j, n Indicates the total number of targets.

The specific steps of the step S4 are:

S41. Determine the number of node feature propagation times T in step S3, control the spread range and depth of node features, and complete the definition of GGNN parameters for the gated graph neural network;

S42. Calculate the initial hidden state of the node, and jointly learn the nodes in the graph structure and the corresponding adjacency matrix in each time step t;

S43. Using the joint learning information at time step t and the node hidden state at the previous time step, according to the node information propagation mechanism of the gated graph neural network GGNN, update the hidden state of the node, and obtain the final hidden state of each node;

S44. Fusing the final hidden state of each node and the initial target category prediction features to obtain the interrelated target category features after joint learning of vision and knowledge.

The expression of the initial hidden state of the node in the step S42 is:

in, Represents the hidden state of the mth node at the initial moment, σ() represents the corresponding activation function, FC() represents the fully connected layer, /> Indicates that the spliced features are mapped to low-dimensional vectors, x _i represents the visual feature vector obtained by feature extraction of the i-th target, p _i represents the target initial category prediction feature of the i-th target, || represents the splicing between features operate;

The expression for joint learning in the step S42 is:

in, Represents the joint learning information of the mth node at the tth time step, N represents the number of categories, /> Indicates the hidden state of the Nth node at time step t-1, G _s indicates the constructed knowledge graph, (G _s ) _m indicates the mth column information of the knowledge graph, and b indicates hyperparameters;

The hidden state calculation process of updating nodes in the step S43 is:

in, Indicates that the mth node controls the forgotten information, σ() represents the corresponding activation function, h ^(t-1) represents the hidden state of the node at t-1 time step, h _m ^(t-1) represents the mth node at t Hidden state for -1 time step, /> Indicates the new information generated by the mth node control, /> Represents the information newly generated by the mth node, tanh() represents the tanh activation function, W ^zT represents the update gate training parameter matrix, U ^zT represents the update gate bias parameter matrix, W ^T represents the reset gate training parameter matrix, U ^T represents Reset the gate bias parameter matrix, W ^rT represents the training parameter matrix representing the new information, U ^rT represents the bias parameter matrix of the new information, ⊙ represents the product of the corresponding elements of the vector, /> Indicates the hidden state of the mth node;

The expression of the target category feature in the step S44 is:

Among them, h _dk represents the initial state of the node in area k, FC() represents the fully connected layer, Indicates the initial node hidden state, /> represents the learned final node state, /> represents rescaling, φ ₀ () represents the aggregation method, p _f represents the interrelated target category features, and p _i represents the target initial category prediction features of the i-th target.

As shown in FIG. 2, in this embodiment, a graph structure is constructed first, and the initial hidden states of nodes are calculated using formulas. And in each time step t, the nodes in the graph structure and the corresponding adjacency matrix are jointly learned. For the i-th target, the visual feature vector x _i is obtained through feature extraction, Represents the mapping of feature vectors to low-dimensional vectors. The target initial category prediction feature p _i of the i-th target is added to form the final node representation of the image.

After the node information propagation is completed, the final hidden state of each region d is obtained. The obtained final hidden state is fused with the initial target category prediction features to obtain fusion-enhanced visual semantic features, that is, interrelated target category features.

The specific steps of the step S5 are:

S51. Use the attention mechanism Transformer to fuse the spatial features of the target bounding box, the visual features of the joint area of the target pair, and the interrelated target category features to obtain the context information of the target pair;

S52. Fusing the visual features of the target pair joint area with the context information of the target pair to obtain relational context features;

S53. According to the relationship context feature, use the softmax function to classify the relationship of each target pair, and complete the visual relationship detection target pair to perform relationship detection.

The calculation formula of the context information of the target pair in the step S51 is:

Q＝Transformer([X, E ^p , E ^f ]; W ^z )

Among them, Q represents the context information of the target pair, that is, the fused features, W ^z represents the training parameters, E ^p represents the embedding vector of target classification prediction, E ^f represents the embedding vector of the target bounding box coordinates, X represents the visual features of the target, Transformer() represents a deep learning model that utilizes an attention mechanism.

In the step S53, the calculation formula for using the softmax function to classify the relationship between each target pair is:

r _ij =q _ij ⊙e _i ⊙u _ij

p(r _k |l,o _i ,o _j )＝softmax(r _ij )

Among them, e _i represents the semantic feature of the target pair, W _z represents the training parameters, r _ij represents the context feature corresponding to target i and target j, and /> The target category features of target i and target j respectively, q _ij represents the context information of the target pair, /> Indicates rescaling, u _ij indicates the visual features of the target pair joint region, ⊙ indicates the product of corresponding elements of the vector, p(r _k |l,o _i ,o _j ) indicates the predicted probability of the target pair relationship classification, r _k indicates the target pair l represents the target image, o _i represents the target i in the target pair, and o _j represents the target j in the target pair.

In this embodiment, the spatial features f _i of the target bounding box obtained in step S1, the visual features u _ij of the joint region of the target pair and the interrelated target category features p _f obtained in step S4 are extracted, and the attention mechanism Transformer is used to learn different The dependencies between features can accurately capture the interaction between them, complete feature fusion, and obtain richer and more accurate context information.

Finally, the visual features of the joint region of the target pair are fused with the contextual information of the target pair to obtain relational contextual features. Visual relationship detection is accomplished by classifying the relationship of each object pair according to the relationship context features.

Claims (9)

1. A visual relationship detection method based on knowledge embedding, characterized in that, comprising the following steps: S1. Input the image, respectively detect the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the target pair joint area; S2. Define the type of prior knowledge, and construct a corresponding knowledge graph for each prior type; S3. Construct the target initial category prediction feature into a graph structure represented by nodes and edges, and represent the knowledge graph as an adjacency matrix corresponding to the graph structure, and represent edge information through the adjacency matrix; S4. Based on the update mechanism of the gated graph neural network GGNN, the nodes of the graph structure and the adjacency matrix are jointly learned with the initial category prediction features of the target to obtain interrelated target category features; S5. According to the spatial features of the target bounding box, the interrelated target category features and the visual features of the target pair joint area, the context information is obtained, and the relationship between each target pair is classified through the softmax function to complete the visual relationship detection. 2. The visual relationship detection method based on knowledge embedding according to claim 1, wherein the specific steps of said step S1 are: S11, input the image, use the feature pyramid network FPN based on the deep residual network ResNet101 as the backbone network of the regional convolutional neural network Faster R-CNN, and extract the visual features of the multi-scale fusion of the image; S12. Pass the visual features through the candidate area network to obtain a target set of interest, and use non-maximum value suppression NMS to obtain candidate areas containing target information; S13. According to the regional feature aggregation method ROIAlign, the features of the candidate regions are uniformly represented, and the initial category prediction features of the target, the spatial features of the target bounding box, and the visual features of the joint area of the target pair are respectively obtained according to the full connection and the softmax function. 3. The visual relationship detection method based on knowledge embedding according to claim 1, wherein the specific steps of the step S2 are: S21. Define the type of prior knowledge through the source of knowledge, wherein the knowledge extracted from the Visual Genome VG dataset is defined as statistical knowledge, and the knowledge extracted from the global vector representation Glove word vector is defined as language knowledge; S22. According to the statistical knowledge, make statistics on the co-occurrence relationship between different categories of objects in the statistical knowledge, obtain the co-occurrence probability distribution table according to the co-occurrence relationship, and measure the cosine similarity of the co-occurrence probability distribution table to obtain internal statistics knowledge map; S23. According to the language knowledge, the language prior knowledge table is obtained through text feature embedding, and the language prior knowledge table is used to measure the cosine similarity to construct an external language knowledge graph; S24. Construct the knowledge graph of the internal statistical knowledge graph and the external language knowledge graph through weighted fusion. 4. the visual relation detection method based on knowledge embedding according to claim 3, is characterized in that, the expression of cosine similarity measure in the described step S22 is: Among them, cos(θ _{i, j} ) represents the cosine similarity measure, δ _i represents the co-occurrence relationship of target i, δ _j represents the co-occurrence relationship of target j, θ _{i, j} represents the similarity between target i and target j, n Indicates the total number of targets. 5. The visual relationship detection method based on knowledge embedding according to claim 1, wherein the specific steps of the step S4 are: S41. Determine the number of node feature propagation times T in step S3, control the spread range and depth of node features, and complete the definition of GGNN parameters for the gated graph neural network; S42. Calculate the initial hidden state of the node, and jointly learn the nodes in the graph structure and the corresponding adjacency matrix in each time step t; S43. Using the joint learning information at time step t and the node hidden state at the previous time step, according to the node information propagation mechanism of the gated graph neural network GGNN, update the hidden state of the node, and obtain the final hidden state of each node; S44. Fusing the final hidden state of each node and the initial target category prediction features to obtain the interrelated target category features after joint learning of vision and knowledge. 6. The visual relationship detection method based on knowledge embedding according to claim 5, characterized in that, the expression of the initial hidden state of the node in the step S42 is: in, Represents the hidden state of the mth node at the initial moment, σ() represents the corresponding activation function, FC() represents the fully connected layer, /> Indicates that the spliced features are mapped to low-dimensional vectors, x _i represents the visual feature vector obtained by feature extraction of the i-th target, p _i represents the target initial category prediction feature of the i-th target, || represents the splicing between features operate; The expression for joint learning in the step S42 is: in, Represents the joint learning information of the mth node at the tth time step, N represents the number of categories, /> Indicates the hidden state of the Nth node at time step t-1, G _s indicates the constructed knowledge graph, (G _s ) _m indicates the mth column information of the knowledge graph, and b indicates hyperparameters; The hidden state calculation process of updating nodes in the step S43 is: in, Indicates that the mth node controls the forgotten information, σ() represents the corresponding activation function, h ^(t-1) represents the hidden state of the node at t-1 time step, h _m ^(t-1) represents the mth node at t Hidden state for -1 time step, /> Indicates the new information generated by the mth node control, /> Represents the information newly generated by the mth node, tanh() represents the tanh activation function, W ^zT represents the update gate training parameter matrix, U ^zT represents the update gate bias parameter matrix, W ^T represents the reset gate training parameter matrix, U ^T represents Reset the gate bias parameter matrix, W ^rT represents the training parameter matrix representing the new information, U ^rT represents the bias parameter matrix of the new information, ⊙ represents the product of the corresponding elements of the vector, /> Indicates the hidden state of the mth node; The expression of the interrelated target category feature in the step S44 is: Among them, h _dk represents the initial state of the node in area k, FC() represents the fully connected layer, represents the initial node hidden state, represents the learned final node state, /> represents rescaling, φ ₀ () represents the aggregation method, p _f represents the interrelated target category features, and p _i represents the target initial category prediction features of the i-th target. 7. The visual relationship detection method based on knowledge embedding according to claim 1, wherein the specific steps of the step S5 are: S51. Use the attention mechanism Transformer to fuse the spatial features of the target bounding box, the visual features of the target pair joint area, and the interrelated target category features to obtain the context information of the target pair; S52. Fusing the visual features of the target pair joint area with the context information of the target pair to obtain relational context features; S53. According to the relationship context feature, use the softmax function to classify the relationship of each target pair, and complete the visual relationship detection target pair to perform relationship detection. 8. The visual relationship detection method based on knowledge embedding according to claim 7, characterized in that, the calculation formula of the context information of the target pair in the step S51 is: Q＝Transformer([X, E ^p , E ^f ]; W ^z ) Among them, Q represents the context information of the target pair, that is, the fused features, W ^z represents the training parameters, E ^p represents the embedding vector of target classification prediction, E ^f represents the embedding vector of the target bounding box coordinates, X represents the visual features of the target, Transformer() represents a deep learning model that utilizes an attention mechanism. 9. The visual relationship detection method based on knowledge embedding according to claim 7, characterized in that, in the step S53, the calculation formula for using the softmax function to classify each target pair is: r _ij =q _ij ⊙e _i ⊙u _ij p(r _k |l,o _i ,o _j )＝softmax(r _ij ) Among them, e _i represents the semantic feature of the target pair, W _z represents the training parameters, r _ij represents the context feature corresponding to target i and target j, and /> The target category features of target i and target j respectively, q _ij represents the context information of the target pair, /> Indicates rescaling, u _ij indicates the visual features of the target pair joint region, ⊙ indicates the product of corresponding elements of the vector, p(r _k |l,o _i ,o _j ) indicates the predicted probability of the target pair relationship classification, r _k indicates the target pair l represents the target image, o _i represents the target i in the target pair, and o _j represents the target j in the target pair.