CN112101387A - Salient element identification method and device - Google Patents
Salient element identification method and device Download PDFInfo
- Publication number
- CN112101387A CN112101387A CN202011020259.4A CN202011020259A CN112101387A CN 112101387 A CN112101387 A CN 112101387A CN 202011020259 A CN202011020259 A CN 202011020259A CN 112101387 A CN112101387 A CN 112101387A
- Authority
- CN
- China
- Prior art keywords
- information
- image
- recognizer
- salient
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a method and a device for identifying a significant element, belonging to the field of mobile communication. The method comprises the following steps: acquiring a first image and first characteristic information associated with the first image; determining a first element identifier according to the first characteristic information; inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information. The embodiment of the application solves the problem that false detection is easily caused by a significant element detection mode in the prior art.
Description
Technical Field
The application belongs to the field of mobile communication, and particularly relates to a method and a device for identifying a salient element.
Background
With the rapid development of mobile communication technology, electronic devices such as smart phones have become an indispensable tool in various aspects of people's life. The functions of various Application programs (APPs) of the electronic equipment are gradually improved, and the functions do not only play a role in communication, but also provide various intelligent services for users, so that great convenience is brought to the work and life of the users.
In terms of the shooting function, a camera of the electronic device is generally performing salient element detection on a shot picture; for example, in a portrait composition scene, a camera first performs salient element detection on people and other subjects in an image to obtain positions of salient elements, and then performs subsequent processing with reference to the positions of the salient elements in a process of recommending composition positions to obtain a better result. The result of the detection of the significant elements directly influences the probability that a significant subject (human or other subjects) is detected by mistake, missed or cut in the process of composing the portrait; therefore, the detection rate of significance detection is improved, and the significance is very important for judging the position relation between the human and other significance subjects.
At present, the salient element detection mainly utilizes traditional image processing, and combines and processes the images based on more primary visual features in the images, such as color, brightness, contrast, edge information and the like, so as to simulate a human visual attention mechanism to obtain the salient elements. In the scheme, some primary first characteristic information of the combined image needs to be artificially designed, and when excessive interference information exists in the image, the method is not universal and is easy to generate false detection, so that the accuracy of identifying the significant elements is influenced.
Disclosure of Invention
The embodiment of the application aims to provide a significant element identification method and a significant element identification device, which can solve the problem that false detection is easily caused by a significant element detection mode in the prior art.
In order to solve the technical problem, the present application is implemented as follows:
in a first aspect, an embodiment of the present application provides a salient element identification method, where the method includes:
acquiring a first image and first characteristic information associated with the first image;
determining a first element identifier according to the first characteristic information;
inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
Optionally, the determining a first element identifier according to the first feature information includes:
acquiring a second element recognizer; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and performing fusion training on each second element recognizer to obtain a first element recognizer.
Optionally, the obtaining the second element identifier includes:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
Optionally, the sub-information includes feature information extracted from the first image, or feature information input by a user.
Optionally, the first image includes a second image displayed on a shooting preview interface of the electronic device or a third image stored in the electronic device.
In a second aspect, an embodiment of the present application further provides a salient element identification apparatus, where the salient element identification apparatus includes:
the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring a first image and first characteristic information associated with the first image;
the determining module is used for determining a first element identifier according to the first characteristic information;
the identification module is used for inputting the first image to the first element identifier to obtain a significant element output by the first element identifier; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
Optionally, the determining module includes:
the identifier obtaining submodule is used for obtaining a second element identifier; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and the fusion sub-module is used for carrying out fusion training on each second element recognizer to obtain the first element recognizer.
Optionally, the identifier obtaining sub-module is configured to:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
Optionally, the sub-information includes feature information extracted from the first image, or feature information input by a user.
Optionally, the first image includes a second image displayed on a shooting preview interface of the electronic device or a third image stored in the electronic device.
In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a program or instructions stored on the memory and executable on the processor, where the processor implements the steps in the salient element identifying method described above when executing the program or instructions.
In a fourth aspect, the present application further provides a readable storage medium, on which a program or instructions are stored, and when the program or instructions are executed by a processor, the program or instructions implement the steps in the salient element identifying method described above.
In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect.
In the embodiment of the application, a first image and first characteristic information associated with the first image are acquired; determining a first element identifier according to the first characteristic information; inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first characteristic information comprises at least one of user characteristic information, geographical position information, time information and scene information; based on the real-time and dynamic first characteristic information, the significance elements of the first image are dynamically identified, the influence of other interference information in the first image is avoided, and the identification accuracy is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive exercise.
Fig. 1 is a flowchart illustrating a salient element recognition method according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating training a second element recognizer according to an embodiment of the present application;
FIG. 3 shows a flow chart of a first example of an embodiment of the present application;
FIG. 4 shows a block diagram of a salient element recognition apparatus provided by an embodiment of the present application;
FIG. 5 shows one of the block diagrams of an electronic device provided by an embodiment of the application;
fig. 6 shows a second block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In various embodiments of the present application, it should be understood that the sequence numbers of the following processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or described herein. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The salient element identification method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.
Referring to fig. 1, an embodiment of the present application provides a salient element recognition method, which is optionally applicable to electronic devices including various handheld devices, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem, as well as various forms of Mobile Stations (MSs), Terminal devices (Terminal devices), and the like.
The method comprises the following steps:
The first image to be recognized is the image to be subjected to the identification of the salient elements; optionally, the first image includes a third image stored in the electronic device or a second image displayed on a shooting preview interface of the electronic device; the third image is an image already stored in the electronic device, such as a picture already taken by the electronic device or a picture received by the electronic device, and the electronic device may perform salient element recognition on the third image during image processing or optimization of the first image. The second image is an image formed by a shooting preview interface of the electronic device, for example, the electronic device performs real-time salient element detection on the shooting preview interface in the process of shooting an image or a video to obtain salient elements in a picture, and then performs composition according to related information of the salient elements.
Specifically, the salient element detection is visual salient detection, and a salient region in an image is extracted by simulating the visual characteristics of a human, wherein the salient region is a region of interest to the human. Objects within the salient region are salient elements. Generally, the human visual system has the ability to quickly search and locate objects of interest when facing natural scenes, and this visual attention mechanism is an important mechanism for people to process visual information in daily life. The significance detection has important application value in the fields of target identification, image video compression, image retrieval, image redirection and the like.
The sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information, that is, the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information. Optionally, the first feature information may be feature information extracted from the first image, or may be feature information input by a user.
The user characteristic information includes user attribute information such as a type of a crowd to which the user belongs, a usage characteristic of the electronic device, and the like. Characteristics of the type of population to which the user belongs, such as age, gender, hobbies, etc.; the use characteristics of the electronic equipment can also reflect the characteristics of the user, such as pictures in an album of the electronic equipment, and the preference characteristics of the user are obtained after clustering; or videos played in the electronic equipment, shopping lists in shopping APP of the user in the electronic equipment and the like can be clustered to obtain the user characteristics.
The geographical location information is used for obtaining the environment where the current electronic equipment (namely the user of the electronic equipment) is located; the time information is used for obtaining factors related to time such as illumination intensity during shooting; the scene information is used to obtain scene information at the time of shooting, such as indoor and outdoor scenes, scenery, sky, night scenes, buildings, and the like.
And judging the salient elements concerned by the user under the current shooting condition through the first characteristic information. For example, if the geographical location information is a scenic spot, the time information is daytime, and the scene information is landscape, the user pays more attention to the overall natural scene, and at this time, the significant element is a scene with a larger pixel in the first image, such as sky or landscape;
or, the geographic location is a downtown area, the photographing time is at night, and when the scene is identified as a night scene, the significant element is a relatively clear and bright object in the environment, such as a luminous logo.
Or, when the geographic location is a mall and the scene is identified as indoor, the salient elements are objects around the user, such as a co-illuminated doll and a food in the hand.
And step 102, determining a first element identifier according to the first characteristic information.
The first element recognizer is used for recognizing a salient element of the first image according to first feature information to obtain the salient element; specifically, the salient element detection is visual salient detection, and a salient region in an image is extracted by simulating the visual characteristics of a human, wherein the salient region is a region of interest to the human. Objects within the salient region are salient elements. Generally, the human visual system has the ability to quickly search and locate objects of interest when facing natural scenes, and this visual attention mechanism is an important mechanism for people to process visual information in daily life. The significance detection has important application value in the fields of target identification, image video compression, image retrieval, image redirection and the like.
The first element recognizer can be trained in advance, or can be obtained by combining the first image with a training sample for training; for example, before step 101, a first element recognizer is trained by a first image and a preset first sample image.
Wherein the first element recognizer is to perform a salient element detection algorithm to detect salient elements in the first image; the salient element detection algorithm is obtained in advance through a machine learning mode, wherein the machine learning mode can be a Convolutional Neural Network (CNN) or a random forest and the like; and training in a machine learning mode to obtain an element recognizer with accuracy meeting the requirement, and identifying the salient elements of the first image.
Specifically, in the identification process, a first element identifier (salient element detection algorithm) performs identification based on first feature information of a first image; the significant elements are obtained, so that the significant elements under more sub-information scenes can be identified, and a significant element main body focused by a user view angle is obtained.
In the embodiment of the application, a first image and first feature information associated with the first image are acquired, and the original image is input to a preset element recognizer to obtain a significant element output by the element recognizer; determining a first element identifier according to the first characteristic information; inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the first feature information comprises at least one of a geographic position feature, a time feature and a scene feature, and the salient elements of the first image are dynamically identified based on the real-time and dynamic first feature information, so that the influence of other interference information in the first image is avoided, and the identification accuracy is improved; the embodiment of the application solves the problem that false detection is easily caused by a significant element detection mode in the prior art.
In an optional embodiment, the determining the first element identifier according to the first feature information includes:
acquiring a second element recognizer; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate; that is, each piece of sub information in the first feature information corresponds to a second element identifier;
and then performing fusion training on each second element recognizer to obtain a first element recognizer.
In this way, the identifiers with the highest identification rate of each sub-information are screened and identified to be fused to obtain a final first element identifier; optionally, fusion can be performed according to a preset fusion algorithm in the fusion process, such as a self-service method (Boosting), a Boosting method (Boosting), and a stacking method (stacking); a self-help method, such as a random forest model, takes the intermediate recognizer as each decision tree in the random forest.
In an alternative embodiment, the first feature information comprises feature information extracted from the first image by the element identifier, or feature information input by a user.
The first feature information includes features extracted from the first image by the element identifier, for example, feature extraction is performed on the first image by a preset feature identifier to obtain first feature information. The first feature information may further include feature information input by a user, for example, when the user inputs the first image, a certain feature is input as the first feature information at the same time, for example, the input scene information is sky.
Optionally, if there is the input first feature information, the first on-element recognizer preferentially performs salient element recognition with the input first feature information; for example, when the first element identifier identifies that the scene information of the first image is outdoors and the user manually inputs the scene information as a landscape, the features input by the user are used as final scene information to meet the identification requirements of the user.
In an optional embodiment, after obtaining the salient elements output by the first element identifier, the method includes:
and according to the salient elements, carrying out image processing on the first image to obtain a target image.
After the identified salient elements, the first image is subjected to image processing, such as composition and other operations in the shooting process, so as to obtain a target image. The image processing may further include processing operations such as target recognition, image video compression, image retrieval, image redirection, and the like, which are not described herein again in this embodiment of the application.
In an optional embodiment, the determining the first element identifier according to the first feature information includes:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image; namely, the second element recognizer is trained in advance;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
Referring to fig. 2, according to the sub information of the first feature information, the process of training the second element identifier corresponding to the sub information is as follows:
The sample image comprises a first image and a second sample image.
The sample image is a sample image used for training a first element identifier (a salient element detection algorithm executed by the first element identifier), and in step 201, the sample image is obtained; to improve the prediction accuracy of the first element recognizer, a large number of second sample images may be acquired for training. The sample image comprises a training sample and a test sample, the training sample is used for training the first element recognizer, and the test sample is used for carrying out accuracy test on the trained first element recognizer to obtain the first element recognizer meeting the accuracy requirement. The second sample image comprises an image and a salient element of the image, namely the salient element in the second sample image is known, and in the training process of the first element recognizer, the salient element in the training sample is used for performing reverse optimization on the first element recognizer; in the testing process, the significant elements in the test sample are used for judging the accuracy of the identification result output by the first element identifier.
Specifically, the salient element detection is visual salient detection, and a salient region in an image is extracted by simulating the visual characteristics of a human, wherein the salient region is a region of interest to the human. Objects within the salient region are salient elements. Generally, the human visual system has the ability to quickly search and locate objects of interest when facing natural scenes, and this visual attention mechanism is an important mechanism for people to process visual information in daily life. The significance detection has important application value in the fields of target identification, image video compression, image retrieval, image redirection and the like.
Wherein the first element identifier is configured to execute a salient element detection algorithm to detect salient elements in the image; the salient element detection algorithm is obtained in advance through a machine learning mode, and the machine learning mode can be a convolutional neural network or a random forest and the like; and training in a machine learning mode to obtain a first element recognizer with accuracy meeting the requirement, and identifying the salient elements of the image.
And 202, training an initial recognizer according to the training sample to obtain a recognizer to be tested.
Inputting the training samples into an initial recognizer to obtain a recognition result, performing reverse optimization on the initial recognizer according to known significant elements, and then iterating the next training sample; and repeating the steps in a circulating way until the identifier to be tested meeting the requirement of the accuracy is obtained. The initial recognizer is determined based on the machine learning algorithm employed, such as an initial random forest model.
Specifically, in the identification process, the first element identifier performs identification based on first feature information of the first image, wherein the first feature information includes at least one of geographical position information, time information and scene information. Optionally, the first feature information may be obtained by the first element identifier through identification of the first image, or may be a feature input to the target identifier simultaneously with the first image.
The geographical location information is used for obtaining the environment where the current electronic equipment (namely the user of the electronic equipment) is located; the time information is used for obtaining factors related to time such as illumination intensity during shooting; the scene information is used to obtain scene information at the time of shooting, such as indoor and outdoor scenes, scenery, sky, night scenes, buildings, and the like.
And judging the salient elements concerned by the user under the current shooting condition through the first characteristic information. For example, if the geographical location information is a scenic spot, the time information is daytime, and the scene information is landscape, the user pays more attention to the overall natural scene, and at this time, the significant element is a scene with a larger pixel in the first image, such as sky or landscape;
or, the geographic location is a downtown area, the photographing time is at night, and when the scene is identified as a night scene, the significant element is a relatively clear and bright object in the environment, such as a luminous logo.
Or, when the geographic location is a mall and the scene is identified as indoor, the salient elements are objects around the user, such as a co-illuminated doll and a food in the hand.
According to the first feature information, distinguishing element identification is carried out on the first image to obtain the distinguishing elements, the distinguishing elements in more scenes can be identified, and a main body of the distinguishing elements which pay attention to the user view angle is obtained.
Optionally, in the process of obtaining the first element recognizer through machine learning training, an ensemble learning manner may be adopted to train multiple intermediate recognizers, which solve the same problem and combine them to obtain a final element detector, so as to obtain a better recognition result. Taking an ensemble learning process as an example of a random forest which is an ensemble learning mode formed by a plurality of decision tree classifiers, training a first element recognizer, and randomly giving each feature value the same weight to obtain an intermediate recognizer; and then continuously classifying and voting the intermediate recognizers according to a large number of known significant elements and the first characteristic information values to finally obtain a group of weight values with the highest accuracy (in the group of weights, each characteristic value corresponds to each weight), and forming the to-be-tested recognizers by the group of weight values and the corresponding intermediate recognizers.
Specifically, when a sample image is selected, a bootstrap (bootstrap) resampling technology is used for repeatedly and randomly drawing k sample images from a sample image set in a replacing manner to generate a new bootstrap sample image set, then k decision trees for classification are generated according to the bootstrap sample image set, and the decision trees are combined together to form a random forest model.
In a random forest, the construction of each tree depends on an independently extracted sample, each tree in the forest has the same distribution, and the classification error depends on the classification capability of each tree and the correlation between the trees. For each feature, each node is divided by a random method, errors generated under different conditions are compared, and the number of the selected features can be determined by detecting the inherent estimation error, the classification capability and the correlation. The classification capability of a single tree may be small, but after a large number of decision trees are randomly generated, the classification capability is inevitably enhanced, and the most possible classification is selected after statistics. And finally obtaining a group of weight values with the highest accuracy as the weight values of the recognizer to be tested through a large amount of classification and regression training.
Testing the identifier to be tested by using a test sample to obtain the first element identifier; and if the special area rate meets the preset requirement, obtaining the first element identifier.
In an alternative embodiment, training an initial recognizer based on the training samples comprises:
inputting the training sample into an initial element recognizer to obtain first characteristic information of the training sample; the first feature information of the training sample is the feature obtained by extracting the feature of the training sample by the initial element recognizer to obtain the initial feature and clustering the initial feature according to a preset clustering algorithm.
In the training process, carrying out feature clustering on the first feature information of the extracted training sample; optionally, the preset clustering algorithm may be K-Means clustering or density-based clustering, etc.; and processing the initial features in a clustering mode to obtain first feature information, so that the identification accuracy of the initial identifier is improved.
In an optional embodiment, the training an initial recognizer according to the training sample to obtain a recognizer to be tested includes:
training an initial recognizer to obtain a plurality of recognizers to be fused according to the training sample; each recognizer to be fused corresponds to one salient element, and the recognizer to be fused is a recognizer which recognizes the highest identification rate of the salient elements;
and training to obtain the recognizer to be tested according to the recognizer to be fused.
In the process of training the recognizers to be tested, firstly, a plurality of intermediate recognizers are obtained according to a training sample and an initial recognizer; selecting a recognizer to be fused with the highest recognition rate for each significant element from the intermediate recognizers; the significant elements are recognized from the training sample by an initial recognizer; and after the recognizer to be fused of each significant element is obtained, performing fusion training on all the recognizers to be fused to obtain the recognizer to be tested.
Optionally, fusion can be performed according to a preset fusion algorithm in the fusion process, such as a self-service method (Boosting), a Boosting method (Boosting), and a stacking method (stacking); the self-help method uses the intermediate recognizer as each decision tree in the random forest, for example, according to the random forest model.
As a first example, referring to fig. 3, fig. 3 shows the main process of a salient element recognition method, comprising the following steps:
Extracting first characteristic information, such as geographical position information, time information and the like, according to information generated by the training sample in the shooting process; meanwhile, some other features, such as scene information, etc., are extracted from the information on the image.
And outputting the significance elements required to be determined by the first characteristic information of the type through a preset clustering method. The first feature information may include geographical location information, temporal information, and scene information, or a combination of the three, to determine the different salient elements.
Geographical location information, where the current environment of the electronic device (i.e., the user of the electronic device) may be obtained; the time information is used for obtaining factors related to time such as illumination intensity during shooting; the scene information is used to obtain scene information at the time of shooting, such as indoor and outdoor scenes, scenery, sky, night scenes, buildings, and the like.
The three characteristics are combined with each other, so that the significant elements concerned by the user in the current photographing scene can be more accurately judged.
Optionally, according to the salient element, a training sample set including the salient element (or the class) is selected, and an intermediate recognizer is obtained through training, wherein the intermediate recognizer has a higher recognition rate for the salient element.
And step 304, fusing the intermediate recognizers to obtain the element recognizer.
And training a plurality of initial recognizers and corresponding first characteristic information to obtain the element recognizer by taking the initial recognizers and the corresponding first characteristic information as reference input. In the training process, first feature information and corresponding significance elements are received, and the significance elements can guide the training of the element recognizer, so that the element recognizer can output the recognition result of specific significance elements according to different user features.
When the element recognizer is actually applied, the first image is received as input, the element recognizer extracts the first characteristic information as reference input, and the salient elements of the first image are output.
In the embodiment of the application, a first image and first characteristic information associated with the first image are acquired; determining a first element identifier according to the first characteristic information; inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first characteristic information comprises at least one of user characteristic information, geographical position information, time information and scene information; on the basis of real-time and dynamic first characteristic information, the salient elements of the first image are dynamically identified, the influence of other interference information in the first image is avoided, and the identification accuracy is improved; the embodiment of the application solves the problem that false detection is easily caused by a significant element detection mode in the prior art.
The salient element recognition method provided by the embodiment of the present application is described above, and the salient element recognition apparatus provided by the embodiment of the present application is described below with reference to the accompanying drawings.
It should be noted that, in the salient element identifying method provided in the embodiment of the present application, the execution subject may be a salient element identifying apparatus, or a control module in the salient element identifying apparatus for executing the salient element identifying method. In the embodiment of the present application, a salient element recognition apparatus executes a salient element recognition method as an example, and the salient element recognition method provided in the embodiment of the present application is described.
Referring to fig. 4, an embodiment of the present application further provides a salient element identification apparatus 400, including:
an obtaining module 401 is configured to obtain a first image and first feature information associated with the first image.
The first image to be recognized is the image to be subjected to the identification of the salient elements; optionally, the first image includes a third image stored in the electronic device or a second image displayed on a shooting preview interface of the electronic device; the third image is an image already stored in the electronic device, such as a picture already taken by the electronic device or a picture received by the electronic device, and the electronic device may perform salient element recognition on the third image during image processing or optimization of the first image. The second image is an image formed by a shooting preview interface of the electronic device, for example, the electronic device performs real-time salient element detection on the shooting preview interface in the process of shooting an image or a video to obtain salient elements in a picture, and then performs composition according to related information of the salient elements.
Specifically, the salient element detection is visual salient detection, and a salient region in an image is extracted by simulating the visual characteristics of a human, wherein the salient region is a region of interest to the human. Objects within the salient region are salient elements. Generally, the human visual system has the ability to quickly search and locate objects of interest when facing natural scenes, and this visual attention mechanism is an important mechanism for people to process visual information in daily life. The significance detection has important application value in the fields of target identification, image video compression, image retrieval, image redirection and the like.
The sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information, that is, the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information. Optionally, the first feature information may be feature information extracted from the first image, or may be feature information input by a user.
The user characteristic information includes user attribute information such as a type of a crowd to which the user belongs, a usage characteristic of the electronic device, and the like. Characteristics of the type of population to which the user belongs, such as age, gender, hobbies, etc.; the use characteristics of the electronic equipment can also reflect the characteristics of the user, such as pictures in an album of the electronic equipment, and the preference characteristics of the user are obtained after clustering; or videos played in the electronic equipment, shopping lists in shopping APP of the user in the electronic equipment and the like can be clustered to obtain the user characteristics.
The geographical location information is used for obtaining the environment where the current electronic equipment (namely the user of the electronic equipment) is located; the time information is used for obtaining factors related to time such as illumination intensity during shooting; the scene information is used to obtain scene information at the time of shooting, such as indoor and outdoor scenes, scenery, sky, night scenes, buildings, and the like.
And judging the salient elements concerned by the user under the current shooting condition through the first characteristic information. For example, if the geographical location information is a scenic spot, the time information is daytime, and the scene information is landscape, the user pays more attention to the overall natural scene, and at this time, the significant element is a scene with a larger pixel in the first image, such as sky or landscape;
or, the geographic location is a downtown area, the photographing time is at night, and when the scene is identified as a night scene, the significant element is a relatively clear and bright object in the environment, such as a luminous logo.
Or, when the geographic location is a mall and the scene is identified as indoor, the salient elements are objects around the user, such as a co-illuminated doll and a food in the hand.
A determining module 402, configured to determine a first element identifier according to the first feature information.
The first element recognizer is used for recognizing a salient element of the first image according to first feature information to obtain the salient element; specifically, the salient element detection is visual salient detection, and a salient region in an image is extracted by simulating the visual characteristics of a human, wherein the salient region is a region of interest to the human. Objects within the salient region are salient elements. Generally, the human visual system has the ability to quickly search and locate objects of interest when facing natural scenes, and this visual attention mechanism is an important mechanism for people to process visual information in daily life. The significance detection has important application value in the fields of target identification, image video compression, image retrieval, image redirection and the like.
The first element recognizer can be trained in advance, or can be obtained by combining the first image with a training sample for training; for example, before step 101, a first element recognizer is trained by a first image and a preset first sample image.
A recognition module 403, configured to input the first image to the first element recognizer, and obtain a salient element output by the first element recognizer; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
Wherein the first element recognizer is to perform a salient element detection algorithm to detect salient elements in the first image; the salient element detection algorithm is obtained in advance through a machine learning mode, wherein the machine learning mode can be a Convolutional Neural Network (CNN) or a random forest and the like; and training in a machine learning mode to obtain an element recognizer with accuracy meeting the requirement, and identifying the salient elements of the first image.
Specifically, in the identification process, a first element identifier (salient element detection algorithm) performs identification based on first feature information of a first image; the significant elements are obtained, so that the significant elements under more sub-information scenes can be identified, and a significant element main body focused by a user view angle is obtained.
Optionally, in this embodiment of the present application, the determining module 402 includes:
the identifier obtaining submodule is used for obtaining a second element identifier; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and the fusion sub-module is used for carrying out fusion training on each second element recognizer to obtain the first element recognizer.
Optionally, in an embodiment of the present application, the identifier obtaining sub-module is configured to:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
Optionally, in this embodiment of the application, the sub information includes feature information extracted from the first image, or feature information input by a user.
Optionally, in this embodiment of the application, the first image includes a second image displayed on a shooting preview interface of the electronic device or a third image stored in the electronic device.
In this embodiment of the present application, the obtaining module 401 obtains a first image and first feature information associated with the first image; the determining module 402 determines a first element identifier according to the first feature information; the recognition module 403 inputs the first image to the first element recognizer, and obtains a saliency element output by the first element recognizer; the sub-information of the first characteristic information comprises at least one of user characteristic information, geographical position information, time information and scene information; on the basis of real-time and dynamic first characteristic information, the salient elements of the first image are dynamically identified, the influence of other interference information in the first image is avoided, and the identification accuracy is improved; the embodiment of the application solves the problem that false detection is easily caused by a significant element detection mode in the prior art.
The salient element recognition apparatus 400 in the embodiment of the present application may be an apparatus, or may be a component, an integrated circuit, or a chip in a terminal. The apparatus 400 may be a mobile electronic device or a non-mobile electronic device. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, and the like, and the embodiments of the present application are not particularly limited.
The salient element recognition device 400 in the embodiment of the present application may be a device 400 having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, and embodiments of the present application are not limited specifically.
The salient element identifying device 400 provided in this embodiment of the application can implement each process implemented by the salient element identifying device 400 in the method embodiments of fig. 1 to fig. 3, and is not described herein again to avoid repetition.
Optionally, as shown in fig. 5, an electronic device 500 is further provided in this embodiment of the present application, and includes a processor 501, a memory 502, and a program or an instruction stored in the memory 502 and executable on the processor 501, where the program or the instruction is executed by the processor 501 to implement each process of the foregoing significant element identification method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.
It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and the non-mobile electronic devices described above.
Fig. 6 is a schematic hardware structure diagram of an electronic device 600 implementing various embodiments of the present application;
the electronic device 600 includes, but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611. Those skilled in the art will appreciate that the electronic device 600 may further comprise a power source (e.g., a battery) for supplying power to the various components, and the power source may be logically connected to the processor 610 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in fig. 6 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.
The processor 610 is configured to obtain a first image and first feature information associated with the first image;
determining a first element identifier according to the first characteristic information;
inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
Optionally, the processor 610 is configured to obtain a second element identifier; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and performing fusion training on each second element recognizer to obtain a first element recognizer.
Optionally, the processor 610 is configured to obtain a preset second element identifier; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
Optionally, the sub-information includes feature information extracted from the first image, or feature information input by a user.
Optionally, the first image includes a second image displayed on a shooting preview interface of the electronic device or a third image stored in the electronic device.
In the embodiment of the application, a first image and first characteristic information associated with the first image are acquired; determining a first element identifier according to the first characteristic information; inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first characteristic information comprises at least one of user characteristic information, geographical position information, time information and scene information; on the basis of real-time and dynamic first characteristic information, the salient elements of the first image are dynamically identified, the influence of other interference information in the first image is avoided, and the identification accuracy is improved; the embodiment of the application solves the problem that false detection is easily caused by a significant element detection mode in the prior art.
It is to be understood that, in the embodiment of the present application, the input Unit 604 may include a Graphics Processing Unit (GPU) 6041 and a microphone 6042, and the Graphics Processing Unit 6041 processes image data of a still picture or a video obtained by an image capturing apparatus (such as a camera) in a video capturing mode or an image capturing mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 607 includes a touch panel 6071 and other input devices 6072. A touch panel 6071, also referred to as a touch screen. The touch panel 6071 may include two parts of a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein. The memory 609 may be used to store software programs as well as various data including, but not limited to, application programs and an operating system. The processor 610 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 610.
The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the above significant element identification method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.
The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and so on.
The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to execute a program or an instruction to implement each process of the above significant element identification method embodiment, and can achieve the same technical effect, and in order to avoid repetition, the description is omitted here.
It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A salient element identification method, comprising:
acquiring a first image and first characteristic information associated with the first image;
determining a first element identifier according to the first characteristic information;
inputting the first image into the first element recognizer to obtain a significant element output by the first element recognizer; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
2. The method according to claim 1, wherein the determining a first element identifier according to the first feature information comprises:
acquiring a second element recognizer; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and performing fusion training on each second element recognizer to obtain a first element recognizer.
3. The salient element recognition method according to claim 2, wherein the obtaining a second element recognizer comprises:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
4. The salient element recognition method according to claim 1, wherein the sub-information comprises feature information extracted from the first image or feature information input by a user.
5. The salient element recognition method according to claim 1, wherein the first image comprises a second image displayed on a shooting preview interface of an electronic device or a third image stored in the electronic device.
6. A salient element recognition apparatus, comprising:
the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring a first image and first characteristic information associated with the first image;
the determining module is used for determining a first element identifier according to the first characteristic information;
the identification module is used for inputting the first image to the first element identifier to obtain a significant element output by the first element identifier; the sub-information of the first feature information includes at least one of user feature information, geographical location information, time information, and scene information.
7. The salient element recognition apparatus of claim 6, wherein the determining module comprises:
the identifier obtaining submodule is used for obtaining a second element identifier; each second element recognizer corresponds to one piece of sub information, and the second element recognizer is a recognizer for recognizing the sub information with the highest recognition rate;
and the fusion sub-module is used for carrying out fusion training on each second element recognizer to obtain the first element recognizer.
8. The salient element recognition apparatus of claim 7, wherein the recognizer obtaining sub-module is configured to:
acquiring a preset second element recognizer; the second element recognizer is obtained by training according to a first sample image;
or
And training a second element recognizer corresponding to the sub information according to the sub information of the first characteristic information.
9. The salient element recognition apparatus according to claim 6, wherein the sub-information comprises feature information extracted from the first image or feature information input by a user.
10. The salient element recognition apparatus according to claim 6, wherein the first image comprises a second image displayed on a shooting preview interface of an electronic device or a third image stored in the electronic device.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011020259.4A CN112101387A (en) | 2020-09-24 | 2020-09-24 | Salient element identification method and device |
| PCT/CN2021/119974 WO2022063189A1 (en) | 2020-09-24 | 2021-09-23 | Salient element recognition method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011020259.4A CN112101387A (en) | 2020-09-24 | 2020-09-24 | Salient element identification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112101387A true CN112101387A (en) | 2020-12-18 |
Family
ID=73756248
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011020259.4A Pending CN112101387A (en) | 2020-09-24 | 2020-09-24 | Salient element identification method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112101387A (en) |
| WO (1) | WO2022063189A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022063189A1 (en) * | 2020-09-24 | 2022-03-31 | 维沃移动通信有限公司 | Salient element recognition method and apparatus |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793925A (en) * | 2014-02-24 | 2014-05-14 | 北京工业大学 | Video image visual salience degree detecting method combining temporal and spatial characteristics |
| CN107622280A (en) * | 2017-09-14 | 2018-01-23 | 河南科技大学 | Modularization prescription formula image significance detection method based on scene classification |
| US20190132520A1 (en) * | 2017-11-02 | 2019-05-02 | Adobe Inc. | Generating image previews based on capture information |
| CN110348291A (en) * | 2019-05-28 | 2019-10-18 | 华为技术有限公司 | A kind of scene recognition method, a kind of scene Recognition device and a kind of electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112101387A (en) * | 2020-09-24 | 2020-12-18 | 维沃移动通信有限公司 | Salient element identification method and device |
-
2020
- 2020-09-24 CN CN202011020259.4A patent/CN112101387A/en active Pending
-
2021
- 2021-09-23 WO PCT/CN2021/119974 patent/WO2022063189A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793925A (en) * | 2014-02-24 | 2014-05-14 | 北京工业大学 | Video image visual salience degree detecting method combining temporal and spatial characteristics |
| CN107622280A (en) * | 2017-09-14 | 2018-01-23 | 河南科技大学 | Modularization prescription formula image significance detection method based on scene classification |
| US20190132520A1 (en) * | 2017-11-02 | 2019-05-02 | Adobe Inc. | Generating image previews based on capture information |
| CN110348291A (en) * | 2019-05-28 | 2019-10-18 | 华为技术有限公司 | A kind of scene recognition method, a kind of scene Recognition device and a kind of electronic equipment |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022063189A1 (en) * | 2020-09-24 | 2022-03-31 | 维沃移动通信有限公司 | Salient element recognition method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022063189A1 (en) | 2022-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111556278B (en) | Video processing method, video display device and storage medium | |
| US11895068B2 (en) | Automated content curation and communication | |
| EP3989104A1 (en) | Facial feature extraction model training method and apparatus, facial feature extraction method and apparatus, device, and storage medium | |
| CN112381104B (en) | Image recognition method, device, computer equipment and storage medium | |
| CN111757175A (en) | Video processing method and device | |
| CN111491187B (en) | Video recommendation method, device, equipment and storage medium | |
| CN113395542A (en) | Video generation method and device based on artificial intelligence, computer equipment and medium | |
| CN112203115B (en) | Video identification method and related device | |
| CN111984749A (en) | Method and device for ordering interest points | |
| CN109948450A (en) | A kind of user behavior detection method, device and storage medium based on image | |
| CN103105924A (en) | Man-machine interaction method and device | |
| CN110263729A (en) | A kind of method of shot boundary detector, model training method and relevant apparatus | |
| CN110347858A (en) | A kind of generation method and relevant apparatus of picture | |
| CN111708944A (en) | Multimedia resource identification method, device, equipment and storage medium | |
| CN112672208A (en) | Video playing method, device, electronic equipment, server and system | |
| CN117671669A (en) | Image recognition method, device, electronic equipment and readable storage medium | |
| CN108052506B (en) | Natural language processing method, device, storage medium and electronic equipment | |
| WO2022063189A1 (en) | Salient element recognition method and apparatus | |
| CN110516689B (en) | Image processing method, image processing device, electronic device and storage medium | |
| CN111722717A (en) | Gesture recognition method and device and computer readable storage medium | |
| CN114220175B (en) | Motion pattern recognition method and device, equipment, medium and product thereof | |
| CN113887534B (en) | Determination method of object detection model and related device | |
| CN112256976B (en) | Matching method and related device | |
| CN112261321B (en) | Subtitle processing method and device and electronic equipment | |
| CN115086759A (en) | Video processing method, video processing device, computer equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |