JP2021081790A - Recognition device and recognition method - Google Patents

Abstract

To provide a technology allowing robust recognition of data.SOLUTION: Features are hierarchically extracted from data acquired from a sensor, and a recognition result to the data is obtained on the basis of the result of the extraction. An acquiring condition of the data from the sensor is controlled on the basis of the result of the extraction.SELECTED DRAWING: Figure 1

Description

The present invention relates to a recognition technique.

A hierarchical calculation method (pattern recognition method based on deep learning technology) represented by Convolutional Neural Network (hereinafter abbreviated as CNN) is attracting attention as a method that enables robust pattern recognition against fluctuations in the recognition target. There is. For example, Non-Patent Document 1 discloses various application examples and implementation examples.

However, even when a powerful calculation method such as CNN is used, sufficient recognition performance may not be obtained depending on the shooting environment (contrast, blur, etc.) of the recognition target.

As a method for dealing with a large change in the shooting environment, Patent Document 1 discloses a method for improving the face detection probability in an image by changing the shooting conditions of the shooting device at predetermined intervals. Further, Patent Document 2 discloses a method of controlling the gain and exposure time of an imaging device based on the result of face detection and reacquiring image data under conditions suitable for attribute recognition processing of the detected person. ..

Japanese Unexamined Patent Publication No. 2014-127999 JP-A-2017-098746

Yann LeCun, Koray Kavacvuoglu and Clement Farabet: Convolutional Networks and Applications in Vision, Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010,

The method disclosed in Patent Document 1 only changes the photographing conditions at predetermined intervals, and does not always obtain the optimum image for pattern recognition. Further, in the method disclosed in Patent Document 2, it is necessary to determine in advance the table for changing the shooting conditions, and it is not possible to determine the changing table for the optimum conditions for various fluctuations in the shooting environment. Have difficulty. In addition, it is not possible to deal with the case where the optimum shooting conditions are different for each region of the same frame image. The present invention provides a technique that enables robust recognition of data.

The uniformity of the present invention is a recognition means for hierarchically extracting features from data acquired from a sensor and acquiring a recognition result for the data based on the extraction result, and from the sensor based on the extraction result. It is characterized in that it is provided with a control means for controlling the data acquisition conditions of the above.

According to the configuration of the present invention, it is possible to provide a technique that enables robust recognition of data.

FIG. 3 is a block diagram showing a more detailed configuration of the recognition device 201. A block diagram showing a configuration example of an image processing system. FIG. 3 is a block diagram showing a configuration example of a recognition device 201 including a logical processing structure of the processing unit 101. The block diagram which shows the structure for realizing the arithmetic processing 303 to 307. A timing chart showing the operation of pattern recognition processing by the image processing system. (A) is a diagram showing an example of a laminated device, and (b) is a diagram showing an example of a logic layer 62. FIG. 6 is a block diagram showing a configuration example of the recognition device 201. The figure which shows an example of the control data corresponding to the logic layer 62. The flowchart which shows the operation of the recognition device 201.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
First, a configuration example of the image processing system according to the present embodiment will be described with reference to the block diagram of FIG. The image processing system according to the present embodiment recognizes the object from the captured image obtained by capturing the object using the imaging device, and images the object from the imaging device based on the features extracted from the captured image for the recognition. Controls image acquisition conditions.

The CPU (Central Processing Unit) 205 executes various processes using computer programs and data stored in a ROM (Read Only Memory) 206 and a RAM (Random Access Memory) 207. As a result, the CPU 205 controls the operation of the entire image processing system, and also executes or controls each process described later as what the image processing system performs.

The ROM 206 stores a startup program and setting data of the image processing system, and a computer program and data for causing the CPU 205 to execute or control each process described later as performed by the image processing system. The computer programs and data stored in the ROM 206 are appropriately loaded into the RAM 207 according to the control by the CPU 205, and are processed by the CPU 205.

The RAM 207 has an area for storing the computer program and data loaded from the ROM 206, and an area for storing the data transferred from the recognition device 201 by the DMAC 208. Further, the RAM 207 has a work area used by the CPU 205 when executing various processes. As described above, the RAM 207 can appropriately provide various areas.

The DMAC (Direct Memory Access Controller) 208 controls the data transfer in the image processing system, and controls, for example, the data transfer between the recognition device 201 and the RAM 207.

Next, the recognition device 201 will be described. The recognition device 201 includes an image pickup device 202, a recognition processing unit 203, and a RAM 204. The image pickup device 202 controls an optical system, a photoelectric conversion device, a signal line and an amplifier for reading an output from a photodiode corresponding to pixels arranged on a sensor surface (sensing region) of the photoelectric conversion device, and the photoelectric conversion device. It has a driver circuit, an A / D conversion unit that converts an analog image signal from the photoelectric conversion device into a digital image signal, and the like. The photoelectric conversion device is a sensor such as a CCD (Charge-Coupled Devices) or a CMOS (Complimentary Metal Oxide Semiconductor).

Light entering from the outside world via an optical system is converted into an analog image signal by a photoelectric conversion device, the analog image signal is converted into a digital image signal by an AD converter, and the digital image signal is recognized as an captured image. It is input to the unit 203.

The recognition processing unit 203 controls the image pickup device 202, recognizes an object included in the captured image acquired from the image pickup device 202, and acquires the position as a recognition result.

The RAM 204 appropriately provides various areas such as a work area used by the recognition processing unit 203 to perform various processes and an area for storing the data transferred by the DMAC 208.

The recognition device 201 performs operations such as imaging and recognition according to an instruction from the CPU 205, and stores the recognition result in the RAM 204. The DMAC 208 transfers the recognition result stored in the RAM 204 to the RAM 207, and the CPU 205 executes various processes based on the recognition result transferred to the RAM 207.

A more detailed configuration of the recognition device 201 will be described with reference to the block diagram of FIG. The processing unit 101 extracts a feature from the captured image acquired from the image pickup device 202, stores the feature (intermediate result of the calculation) in the memory 103, and performs the next calculation using the stored feature. By repeating the process, features are hierarchically extracted from the captured image. Then, the processing unit 101 stores the recognition result (recognition result for the captured image) based on the feature finally extracted from the captured image in, for example, the RAM 204. Further, the processing unit 101 uses the features hierarchically extracted from the captured image to generate control data for controlling the acquisition conditions of the captured image from the imaging device 202, and the generated control data is used in the processing unit 105. Output to.

The processing unit 105 controls the image pickup device 202 based on the control data from the processing unit 101 to control the acquisition conditions for the processing unit 101 to acquire the captured image from the image pickup device 202.

For example, the processing unit 105 determines the gain for the signal after photoelectric conversion, the charge accumulation time (exposure time) of the photoelectric conversion device (photodiode, etc.), the A / D conversion characteristics of the A / D conversion unit, and the like according to the control data. To control. In the present embodiment, when the sensor surface of the photoelectric conversion device is divided into a plurality of regions, each divided region is referred to as a block, and the processing unit 105 controls acquisition conditions on a block-by-block basis.

With the practical application of semiconductor laminated mounting technology in recent years, it has become possible to realize read control in block units or pixel units by laminating and mounting control logic on the sensor surface. An example of a laminated device applicable to this embodiment is shown in FIG. 6 (a). For the sensor layer 61 (corresponding to the photoelectric conversion device) on which the photoelectric conversion element is mounted, the logic layer 62 (corresponding to the processing unit 105) on which the read control logic is mounted, the large-scale memory and the memory layer 63 on which the control unit is mounted. (Corresponding to memory 103) is stacked. Signals are transmitted between each layer by means of penetrating vias or the like.

An example of the logic layer 62 is shown in FIG. 6 (b). The logic layer 62 is provided with control circuits ct (1,1) to ct (n, n) corresponding to each block in the sensor layer 61, and the control circuit ct is from a block corresponding to the control circuit ct. Controls the reading of data. FIG. 6B shows a configuration for controlling acquisition conditions (gain, exposure time, etc.) for each of n × n blocks on the sensor surface. That is, the imaging characteristics can be controlled for each of n × n partial images in the image.

In this embodiment, the processing unit 101 is also mounted on the logic layer 62 and the memory layer 63. By stacking and mounting on the sensor layer 61, control data can be fed back to the processing unit 105 with less delay. When the shooting environment or the target changes at high speed, it is desired to control the imaging device 202 with a smaller image frame delay.

The control unit 102 controls the operation of the processing unit 101 and the processing unit 105 of the recognition device 201. A configuration example of the recognition device 201 including the logical processing structure of the processing unit 101 will be described with reference to the block diagram of FIG. The processing unit 101 has a recognition network 302 and a sensor control network 313.

The recognition network 302 is a hierarchical neural network that recognizes a specific object in the captured image 301 captured by the imaging device 202 and outputs a recognition result indicating the position of the recognized specific object. In the present embodiment, the recognition network 302 has five layers. It will be described as being CNN.

The sensor control network 313 is a hierarchical neural network that generates control data from a feature map generated in the recognition network 302, and will be described as a two-layer CNN in the present embodiment.

First, the recognition network 302 will be described. Each of 303 to 307 represents a calculation process including a convolution operation, an activation function operation, a pooling operation, and the like, and is a process that can be implemented with the configuration shown in FIG. The configuration shown in FIG. 4 will be described later. The feature map (Fature Map) 308 to 311 is a feature map called an intermediate layer in the CNN, and the feature map 312 is a feature map called the final layer in the CNN. Each feature map is two-dimensional data hierarchically extracted from the captured image 301 and is stored in the memory 103.

The feature map 308 is a feature map obtained by arithmetic processing 303 for the captured image 301, and the feature map 309 is a feature map obtained by arithmetic processing 304 from the feature map 308. The feature map 310 is a feature map obtained from the feature map 309 by the calculation process 305, and the feature map 311 is a feature map obtained from the feature map 310 by the calculation process 306. The feature map 312 is a feature map obtained by arithmetic processing 307 from the feature map 311 and is also a recognition result for the captured image 301.

Here, the details of the two-dimensional CNN calculation performed by the recognition network 302 on the captured image 301 will be described. When the kernel (coefficient matrix) size of the convolution operation is volumeSize × lowSize and the number of feature maps in the previous layer is L, one feature map is calculated by the product-sum operation as shown in the following equation (1).

input (x, y): Reference pixel value in two-dimensional coordinates (x, y) output (x, y): Calculation result in two-dimensional coordinates (x, y) weight (collect, low): Two-dimensional coordinates ( Weight coefficient at x + volume, y + low) L: Number of feature maps in the previous layer volumeSize: Horizontal size of 2D convolution kernel lowSize: Vertical size of 2D convolution kernel In 2D CNN calculation, multiple The feature map is calculated by repeating the product-sum operation while scanning the convolution kernel in pixel units and performing non-linear conversion (activation processing) of the final product-sum operation result. In addition, the generated feature map may be reduced by pooling processing and referred to in the next layer. A plurality of feature maps 308 to 312 exist in one corresponding hierarchy, and feature maps of different characteristics are generated corresponding to different weight coefficient groups.

The weighting coefficient used in the two-dimensional CNN operation is a data set determined by prior learning. The weight coefficient is collected by learning in advance with a learning device (general-purpose computer, etc.) outside the image processing system using learning data and teacher data (data indicating the correct answer) by a learning method such as back propagation. Keep it.

Next, the configuration for realizing the arithmetic processes 303 to 307 will be described with reference to the block diagram of FIG. The data buffer 401 is for acquiring all or a part of the data of the feature map of the previous layer (input (x, y) in the equation (1)) which is the reference data of the convolution operation from the memory 103 and buffering it. It is a memory circuit.

The multiplier 402 and the cumulative adder 403 are circuits that perform multiplication and cumulative addition, respectively, and the calculation of equation (1) is performed by the multiplier 402 and the cumulative adder 403. The data buffer 404 is a memory circuit that reads out all or a part of the weighting coefficient (weight (collect, low) in the equation (1)) obtained by learning in advance from the memory 103 in a predetermined unit and buffers it. The multiplier 402 performs a multiplication operation using the weighting coefficient stored in the data buffer 404.

The activation processor 405 is a circuit that performs an operation of applying a non-linear function such as ReLU (Rectifier Line Unit, Rectifier) to the convolution operation result (output (x, y)) shown in the equation (1).

The pooling processor 406 is a circuit that reduces the feature map by using a spatial filter such as a maximum value filter and stores the reduced feature map in the memory 103. When the pooling process is not performed, the calculation result by the activation processor 405 is stored in the memory 103. When the pooling process is performed, the process result by the pooling processor 406 is stored in the memory 103. The feature map stored here is the feature map of the current hierarchy. When the calculation of the feature map of the current layer is completed, the data is used as the feature map of the previous layer for the calculation of the feature map of the next layer. In this way, the feature maps of a plurality of layers are calculated while sequentially referring to the feature maps stored in the memory 103. The control unit 102 controls the operation of each functional unit shown in FIG. 4 to realize a hierarchical feature extraction process (two-dimensional CNN arithmetic process).

By repeating feature extraction over a plurality of layers in this way, CNN realizes a recognition process that is robust to fluctuations in the identification target. According to the feature extraction result of each layer, the existence of a desired object in the captured image 301 is determined by the arithmetic processing 307 which is the arithmetic processing in the final layer. The feature map 312 of the final layer expresses the recognition result. The recognition result represented by the feature map 312 is output as, for example, a reliability map expressing the existence probability of a desired object in the captured image 301 as two-dimensional information. The arithmetic processing 307, which is the arithmetic processing in the final layer, may be implemented by a fully connected neural network or a linear discriminator instead of the convolution operation described above.

Further, the feature maps 308 to 311 of each layer represent the feature extraction results for the captured image 301. In general, the feature map of the lower layer (the layer closer to the layer into which the captured image 301 is input) shows low-level features such as edges, and the feature map of the upper layer (the layer closer to the recognition result) has a high degree of abstraction. Show the features. The characteristics of each feature map differ depending on the target of pattern recognition and the learning method.

Next, the sensor control network 313 will be described. The sensor control network 313 is an arithmetic network that returns control data to the processing unit 105. Each of 314 and 315 represents an arithmetic process including a convolution operation, an activation function operation, a pooling operation, and the like, similar to those of the arithmetic processes 303 to 307, and is a process that can be implemented with the configuration shown in FIG.

The sensor control network 313 according to the present embodiment receives the feature map 308 of the lower layer in the recognition network 302 as an input, and generates control data as regression data from the feature map 308. By sharing the feature map with the recognition network 302, it is expected that the regression performance will be improved and learning will be facilitated, and the overall calculation cost can be reduced. Further, in the present embodiment, since the sensor control network 313 is configured with a network structure similar to the recognition network 302, the configuration shown in FIG. 4 can be shared between the recognition network 302 and the sensor control network 313. As a result, it is not necessary to provide a circuit for generating control data separately from the circuit for recognition.

The feature map 316 is a feature map obtained from the feature map 308 by the calculation process 314, and the feature map 317 is a feature map obtained from the feature map 316 by the calculation process 315. The feature map 317 is returned to the processing unit 105 as control data.

The control data is data that specifies acquisition conditions corresponding to the spatial positions of the pixels (imaging elements) arranged on the sensor surface, and corresponds to, for example, the designation of the gain and exposure time of the pixels corresponding to the positions in the feature map. It becomes data. The control data may be a single feature map when the control target is one type and is controlled by a scalar value. When there are a plurality of control targets or the control parameter is vector data, the control data is a plurality of feature maps.

FIG. 8 shows an example of control data corresponding to the logic layer 62 of FIG. 6 (b). The control data shown in FIG. 8 is composed of a plurality of feature maps, and rg (n, n) in one of the feature maps 218 is an acquisition condition corresponding to a block corresponding to ct (n, n). Represents. In FIG. 8, the value of the acquisition condition is represented by a shade, and the value of the acquisition condition corresponds to, for example, a gain.

As for the sensor control network 313, as in the recognition network 302, the weighting coefficient is acquired in advance by learning with a computer or the like outside the image processing system. The learning here also uses the teacher data in the same manner as the learning of the recognition network 302, and performs learning in cooperation with the recognition network 302. Learning is further carried out by using back propagation or the like in consideration of the characteristics of the sensor.

The processing unit 105 controls each pixel in the photoelectric conversion device of the imaging device 202 according to the control data returned by the sensor control network 313 (for example, the gain of the data from the pixel on the sensor surface is determined according to the value of the corresponding acquisition condition. To control). As a result, the processing unit 101 can acquire an captured image suitable for the recognition process from the imaging device 202. The captured image obtained here is different from an image for human observation to understand and appreciate the contents, and is an image suitable for improving the accuracy of the recognition process.

In the present embodiment, the arithmetic processing 314 in the sensor control network 313 includes a pooling processing, and as a result, a feature map 316 obtained by reducing the feature map 308 is obtained, and the arithmetic processing 315 targets the feature map 316. Is done. Therefore, the size of the control data (feature map 317) is smaller than the size of the captured image 301. That is, the acquisition condition is controlled for each block having a plurality of pixels as a unit. The pooling ratio and the like are set in advance in consideration of the block size that can be controlled by the processing unit 105.

The operation of the pattern recognition process by the image processing system will be described with reference to the timing chart of FIG. In FIG. 5, the “recognition network X” represents the Xth operation of the recognition network 302, and the “sensor control network Y” represents the Yth operation of the sensor control network 313. FIG. 5 shows how the recognition network 302 and the sensor control network 313 process each of the captured images for three frames.

The recognition network 1 and the sensor control network 1 are executed in parallel. The control data 507 is obtained as a processing result by the sensor control network 1, and the recognition network 2 is executed for "the captured image of the next frame obtained from the imaging device 202 under the acquisition conditions according to the control data 507". To. The sensor control network 2 is executed in parallel with the recognition network 2.

The control data 508 is obtained as a processing result by the sensor control network 2, and the recognition network 3 is executed for "the captured image of the next frame obtained from the imaging device 202 under the acquisition conditions according to the control data 508". To. The sensor control network 3 is executed in parallel with the recognition network 3.

The control data 509 is obtained as a processing result by the sensor control network 3, and the recognition network 4 is executed for "the captured image of the next frame obtained from the imaging device 202 under the acquisition conditions according to the control data 509". To.

The operation of the recognition device 201 will be described with reference to the flowchart of FIG. Since the details of the processing in each step of FIG. 9 are as described above, they will be briefly described here.

In step S901, the recognition network 302 takes the captured image from the imaging device 202 as an input and performs recognition processing on the captured image by hierarchically extracting features from the captured image.

When the feature map of a specific hierarchy is obtained in the hierarchical feature extraction in step S901 (when the feature map 308 is obtained in the example of FIG. 3), the process of step S902 is started. In step S902, the sensor control network 313 receives the feature map of a specific layer as an input, performs the above processing, generates control data, and outputs the generated control data to the processing unit 105. Then, in step S903, the processing unit 105 controls the acquisition conditions of the captured image from the imaging device 202 for each block based on the control data acquired from the sensor control network 313.

In step S904, the control unit 102 determines whether or not the end instruction has been received. For example, the control unit 102 may acquire the end instruction input by the user by operating the operation unit (not shown), or the end instruction issued by the CPU 205 when the CPU 205 detects that a specific condition is satisfied. May be transferred to the control unit 102 by the DMAC 208.

As a result of this determination, when the control unit 102 receives the end instruction, the process according to the flowchart of FIG. 9 ends, and when the control unit 102 does not receive the end instruction, the process follows the flowchart of FIG. Return to the beginning of.

In this way, the sensor control network sequentially sets shooting conditions suitable for recognition according to changes in the situation of the shooting target, and the recognition network executes highly accurate recognition processing according to the shooting conditions. As described above, according to the present embodiment, it is possible to acquire the optimum image for recognition according to the shooting environment, and it is possible to realize a recognition technique with high recognition accuracy.

[Second Embodiment]
In each of the following embodiments including the present embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, the same as the first embodiment. A configuration example of the recognition device 201 according to the present embodiment will be described with reference to the block diagram of FIG. The configuration shown in FIG. 7 is a configuration in which the image correction processing unit 706 is added to the configuration shown in FIG. Further, the captured image output from the image pickup device 202 is input not only to the processing unit 101 but also to the image correction processing unit 706, and the control data output from the processing unit 101 is sent not only to the processing unit 105 but also to the image correction processing unit 706. Is also entered.

In the first embodiment, the captured image output by the imaging device 202 is read out as an image suitable for the recognition process, and is therefore not preferable as an image observed by a human. For example, in a surveillance camera or the like, there are cases where a person confirms a detected object after the fact.

In the present embodiment, the image correction processing unit 706 observes the captured image output from the imaging device 202 (that is, the captured image suitable for the recognition process) based on the control data output from the processing unit 101. Converts to a natural image. Image conversion can be performed according to a predetermined algorithm based on the gain and exposure time, which are the shooting conditions represented by the control data (it can be handled by extending the image processing called development processing). Further, the image correction processing unit 706 may also use a method such as converting an image based on the learning data by using CNN or the like. In that case, the configuration shown in FIG. 4 can be used as it is, and no additional circuit is required in the configuration.

The output destination of the captured image converted by the image correction processing unit 706 is not limited to a specific output destination, and may be a display unit inside or outside the image processing system, or may be a memory inside or outside the image processing system. .. As described above, according to the present embodiment, it is possible to acquire an image suitable for pattern recognition and output an image observable by a person.

[Third Embodiment]
In the first embodiment, a configuration using a two-dimensional image sensor has been described as an example, but the sensor is not limited to the two-dimensional image sensor, and various sensors having different numbers of dimensions and modalities of the data to be sensed are used. It may be the configuration that was used. Examples of such a sensor include a microphone and a radio wave sensor. That is, in the first embodiment, features are hierarchically extracted from the data acquired from the sensor, a recognition result for the data is acquired based on the extraction result, and the sensor is used based on the extraction result. This is just an example of a configuration in which the data acquisition conditions of the above are controlled.

Further, in the first embodiment, the case where the acquisition condition is controlled in block units has been described, but the control unit is not limited to blocks, and may be pixels or the entire sensor surface. When the acquisition conditions are controlled in units of sensor surfaces, the result of passing the feature map of the final layer of the sensor control network 313 through a linear discriminator may be used as control data, or the feature map is subjected to global pooling processing. The result may be used as control data.

Further, in the first embodiment, the sensor control network 313 inputs the feature map of the lower layer of the recognition network 302, but the feature map to be input is not limited to the feature map of the lower layer. For example, the sensor control network 313 may input the feature map of the upper layer of the recognition network 302, or may input the feature map of the layer selected from the feature map of each layer of the recognition network 302. The selection may be made by the control unit 102, or the user may operate an operation unit (not shown), and the selection is not limited to a specific form.

Further, the hierarchical structure of the recognition network 302 and the sensor control network 313 (the number of layers, the number of feature maps in the hierarchy, etc.) can be appropriately changed according to the recognition target, the control target, and the like.

Further, the sensor control network 313 may input an output (captured image) from the imaging device 202 instead of the feature map of the recognition network 302. In that case as well, when learning the sensor control network 313, learning is performed using the recognition network 302.

Further, in the first embodiment, the case where the reliability of pattern recognition and control data are generated in the final layer has been described, but the present invention is not limited to this, and for example, the reliability of pattern recognition is directly referred to by the feature map of the intermediate layer. The degree and control data may be generated.

Further, in the first embodiment, the image processing system for detecting the position of the object in the captured image has been described, but the task performed by the image processing system is not limited to this, and for example, recognition of the attribute of the object in the captured image. Or, various recognitions such as recognition of the contents of the captured image may be performed.

Further, in the first embodiment, CNN is used to extract features hierarchically, but the present invention is not limited to this. That is, features may be extracted hierarchically using various other hierarchical methods such as Multi Layer Perceptron, Restricted Boltzmann Machines, and Capsule Network. In addition, a recursive method such as Recursive Neural Network may be used.

In the first embodiment, the case where the configuration shown in FIG. 4 is implemented by hardware has been described, but some configurations, for example, a multiplier 402, a cumulative adder 403, an activation processor 405, and a pooling processor 406 have been described. May be implemented by software (computer program). In this case, this software is stored in the ROM 206, transferred to the RAM 204 by the DMAC 208, and executed by the control unit 102, so that the functions of the corresponding functional units can be realized.

Further, in the first embodiment, the laminated device shown in FIG. 6 is applied, but the mounting function on each laminated can be in various forms in consideration of cost and performance. Further, in the case where the delay of the read control is not a problem, the recognition network 302 and the sensor control network 313 may be mounted on different devices without being stacked.

Further, in the first embodiment, a case applied to pure recognition processing has been described, but a method applying a deep learning technique proposed in recent years not only recognizes a specific pattern but also transforms / transforms the pattern. A method to be used for such purposes has also been proposed. Therefore, the first embodiment can be applied to these methods.

It should be noted that some or all of the above-described embodiments may be used in combination as appropriate. In addition, a part or all of each of the above-described embodiments may be selectively used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101: Processing unit 102: Control unit 103: Memory 105: Processing unit 202: Imaging device

Claims (13)

A recognition means that extracts features hierarchically from the data acquired from the sensor and acquires the recognition result for the data based on the extraction result.
A recognition device including a control means for controlling acquisition conditions of data from the sensor based on the result of the extraction. The recognition device according to claim 1, wherein the recognition means hierarchically extracts features from data acquired from the sensor using a hierarchical neural network. The control means generates control data for controlling the acquisition condition based on a feature map of a part of the hierarchy in the hierarchical neural network, and controls the acquisition condition based on the control data. The recognition device according to claim 2. The recognition device according to claim 3, wherein the control means generates the control data from a feature map of a part of the layers by using a hierarchical neural network. The recognition device according to claim 4, wherein the hierarchical neural network used by the recognition means and the hierarchical neural network used by the control means are convolutional neural networks. The recognition device according to any one of claims 1 to 5, wherein the control means controls the acquisition conditions for each divided region in which the data sensing region of the sensor is divided. The recognition device according to any one of claims 1 to 6, wherein the sensor is an image pickup device, and the data is an image captured image captured by the image pickup device. In addition
The recognition device according to claim 7, further comprising a correction means for correcting the captured image based on data for controlling the acquisition conditions. The recognition device according to claim 7, wherein the imaging device includes a photoelectric conversion device, and the control means controls the charge accumulation time of the photoelectric conversion device. The recognition device according to claim 7, wherein the imaging device includes a photoelectric conversion device, and the control means controls a gain for a signal after photoelectric conversion by the photoelectric conversion device. The imaging device includes an A / D conversion unit that converts an analog image signal into a digital image signal, and the control means controls the characteristics of A / D conversion by the A / D conversion unit. Item 7. The recognition device according to item 7. It is a recognition method performed by the recognition device.
A recognition step in which the recognition means of the recognition device hierarchically extracts features from the data acquired from the sensor and acquires the recognition result for the data based on the extraction result.
A recognition method, wherein the control means of the recognition device includes a control step of controlling acquisition conditions of data from the sensor based on the result of the extraction. A computer program for causing the computer of the recognition device to function as each means of the recognition device according to any one of claims 1 to 11.

Images