DE102021105869B4 - CORRECTION AND CALIBRATION OF A SPECTRAL SENSOR OUTPUT - Google Patents

Abstract

A spectral sensor system comprising:a plurality of optical sensors (100) arranged on an integrated circuit and forming a spectral sensor (10), the plurality of optical sensors (100) arranged in an array (170);an interface between the plurality of optical sensors (100) and a first processing device configured to transmit information therebetween;a plurality of sets of optical filters (110, 120, 130, 160) configured as a layer arranged on the plurality of optical sensors (100), wherein a set of optical filters (110, 120, 130, 160) of the plurality of sets of optical filters (110, 120, 130, 160) comprises a plurality of optical filters (110, 120, 130, 160), each optical filter (110, 120, 130, 160) of the plurality of optical filters (110, 120, 130, 160) is configured to transmit light (180) in a different wavelength range;a memory configured to interface and communicate with the first processing device, the memory configured to store calibration data (250, 350) associated with the plurality of sets of optical sensors (100);a spectrometer model (510) reflecting the non-ideality of the spectral sensor (10);a second processing device comprising an artificial neural network (370) configured to correct a spectral response generated by the plurality of optical sensors (100); based on an error (530) determined for artificial spectra (560) and input to the artificial neural network (370) to generate coefficients as weights of the artificial neural network (370) that reflect the difference between the artificial spectra (560) and the spectrometer model (510); andan interface between the first processing device and the second processing device configured to transmit information therebetween.

Description

BACKGROUND OF THE INVENTION

WO 2015/015493 A2 , CA 2 954 625 A1 and WO 2018/056976 A1 describe conventional spectrometers or spectrally resolving imaging systems.

TECHNICAL FIELD OF THE INVENTION

This invention relates generally to spectroscopy and, more particularly, to spectral correction for spectral sensors using interference-based filters.

Spectroscopy devices have proven useful for applications in various industries, including, for example, health, biometrics, agriculture, chemistry, and fitness. In general, spectroscopy devices work by detecting and/or sensing incident light with respect to multiple wavelength ranges and extracting spectral information. It has been shown that interference-based filters such as Fabry-Pérot filters can provide controlled wavelengths of light when coupled with spectral sensors.

As is further known, light passing through interference-based filters is subject to various non-ideal conditions as well as non-ideal sensor performance, each of which can negatively affect the performance of a particular spectroscopy instrument.

DESCRIPTION OF RELATED TECHNOLOGY

BRIEF DESCRIPTION OF THE MULTIPLE VIEWS OF THE DRAWING(S)

• 1A provides a top-down illustration of an exemplary optical sensor overlaid with filters in accordance with the present invention;
• 1B shows a side view of an exemplary optical sensor overlaid with filters according to the present invention;
• 2 shows the example filter responses of a CMOS sensor overlaid with filters;
• 3 provides a comparison of raw output data and the ideal output data of an example spectral sensor;
• 4 is a schematic block diagram of an embodiment of a spectral sensor according to the present invention;
• 5A provides a schematic block diagram of an embodiment of a spectral sensor incorporating an artificial neural network according to the present invention;
• 5B provides a schematic block diagram of another embodiment of a spectral sensor incorporating an artificial neural network according to the present invention;
• 6A provides a simplified representation of an artificial neural network incorporated into a spectral sensor according to the present invention;
• 6B provides a simplified representation of a multilayer artificial neural network according to the present invention;
• 7A is a logic diagram illustrating an exemplary method for correcting the raw spectral output into corrected spectra in accordance with the present invention;
• 7B is a logic diagram illustrating an exemplary method for adaptively correcting the spectral output from a sensor array in accordance with the present invention;
• 7C provides a logic diagram illustrating another exemplary method for adaptively correcting the spectral output from a sensor array in accordance with the present invention;
• 8A provides an illustration of an exemplary spectral sensor in which a sweep of monochromatic light and input power is provided in accordance with the present invention;
• 8B provides an exemplary spectral sensor incorporating an artificial neural network according to the present invention;
• 8C provides an illustration of an example of a physical scene provided to a spectral sensor and output to a trained artificial neural network in accordance with the present invention;
• 8D provides an exemplary spectral sensor in which metadata of a physical scene is provided to the artificial neural network according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In various embodiments, spectral sensors are combined with interference filters to obtain spectral information about a scene and/or light source. Interference-based filters such as Fabry-Pérot filters may exhibit a non-ideal filtering response to an underlying sensor. For example, the filtering response may include crosstalk between filters, unwanted second-order responses to incident light, angular dependencies to incident light, and incident light may include light from sources that are not intended to be evaluated. Sensors themselves may also exhibit non-ideal performance, including nonlinearities, crosstalk, electronic noise, etc. 1A shows, from top to bottom, an example integrated optical sensor 100 overlaid with filters 110, 120, and 130 optimized for one of three spectral bands. As shown, filters 110, 120, and 130 repeat as an array across the surface of optical sensor 100. Alternatively, filters 110, 120, and 130 could repeat using a different pattern or even in a random pattern to route filter responses to sensors underlying the filter array. In one example (not shown), spectral bands greater than 3 could be used to overlay sensors in any practical orientation as desired. In one embodiment, optical sensor 100 is an example of a spectral sensor useful for diffuse optical spectroscopy, where arrays of spectral filters are associated with optical sensors to provide diffuse spectral sensing. 1B shows a side view of an exemplary optical sensor overlaid with a filter array. In the example, incident light 180 is directed through the filter assembly 160 to the sensor array 170.

Ideally, the filters of the integrated optical sensor in Figures 1A and 1B would only pass incident light within the desired spectral wavelengths to the underlying sensor, so that an ideal sensor could then output filter responses that accurately represent the spectrum of the incident light. In practice, neither the interference filters nor the sensors are ideal, so if the output of a given optical sensor, such as the one in Figures 1A and 1B shown, the resulting spectrum is unlikely to accurately represent the spectrum of the incident light. Indeed, the resulting spectrum may have a non-linear relationship with the intensity of the input light, the spectrum of an object being measured, and/or the illumination spectrum of the input light. In addition, manufacturing-related errors such as manufacturing tolerances and uniformity between integrated optical sensors, such as integrated optical sensor 100, may contribute to non-ideal performance.

2 shows the filter responses of a CMOS sensor (in this case using Fabry-Pérot filters) that exhibit crosstalk from neighboring sensors. In an ideal sensor, the filter response of a particular sensor pixel, such as that of pixel #13, would come only from pixel #13, without contributing to other pixels within the array. As shown, neighboring pixels #8, #12, #14, and #18 show at least an undesirable response from the wavelength centered on pixel #13, along with other nonidealities. Reconstructing a “clean” output spectrum (one that adequately represents the spectrum of light incident on the optical sensor) is not trivial, even if the filter response shapes shown are known. In the plot of 2 the responses do not reflect the perfect "Fabry-Pérot" responses, and the generation of crosstalk from the neighboring filters is problematic and contributes to the response error. Considering that each pixel can have non-ideal responses and crosstalk errors, it is easy to understand that correcting and/or calibrating a given optical sensor can become extremely difficult and/or inaccurate. In addition, material properties of the filters themselves, quality differences, and part-to-part differences add additional complexity.

3 provides a representation of the output data 190 of a reflectance measurement (filter response) from approximately 740 to 1020 nm of an exemplary optical sensor, along with an ideal or "corrected" spectrum 192 for the measurement. In the example, the raw output data cannot be adjusted with a complex correction exercise to convert it to a "real" spectrum (i.e., a spectrum representative of the spectrum of the incident light).

In an example, the raw power of a spectral sensor ORAW can be measured as a matrix of [1xN] values. It can be corrected by a matrix multiplication so that: O=O _RAW xC where O is the corrected spectrum and C [NxM] is a correction matrix where M is the desired output wavelength response(s) (in a linear correction step). C is a "correction matrix" constructed from measured and/or derived knowledge of the sensor and filters, or factory measurement/calibration of the sensor's filter characteristics, or a combination of both.

In one example, O _RAW is constructed from N (eg with N = 64 in a 64-channel spectrometer built from N light-sensitive elements and N filters over the elements) values from the filter outputs. In the example of 3 the 64 outputs are limited to the sensor outputs and represent the fully integrated response of each individual filter on each photodiode. Referring to 1A a spectral sensor may include a plurality of optical sensors having a plurality of sets of optical filters configured as a layer having a top surface and a bottom surface located on the plurality of optical sensors. In the example, a set of optical filters of the plurality of sets of optical filters comprises a plurality of optical filters, each optical filter of the plurality of optical filters being configured to transmit light in a different wavelength range. In one example, a plurality of optical filters configured to transmit light in the same wavelength range may be provided in an array such that, in a given array, each wavelength range is represented redundantly. Numerous layout options for the optical filters are possible. In one particular example, the optical filters of each set of optical filters may be arranged in a pattern that repeats across a sensor array. In another example, each set of optical filters may be configured in multiple different patterns within the same sensor array. Accordingly, due at least in part to the non-idealities discussed with reference to the 1A , 1B and 2 discussed, each of the plurality of optical filters configured to transmit light in the same wavelength range behaves differently.

Referring again to 3 a given sensor output does not necessarily contain the full influence of the spectral responses from each filter on the sensor, and this behavior (as in 2 shown above) is extremely complex and cannot be adequately captured in just 64 sampled data points. In addition, the other non-idealities associated with the sensor are not captured in the output, including, for example, its inherent non-linearities and the non-linearities that affect crosstalk, among other things. Accordingly, providing a single correction matrix that can provide a corrected spectrum over all conditions of input power, illumination spectral profile, and scene spectral profile can be difficult, if not mathematically impossible. In one example, the correction matrix C from the above equation is "optimized" using a least squares approach to provide desired and/or expected spectral profiles in spectral measurements. More complex mathematical correction matrices can be created, but as shown, they can present an extremely complex mathematical problem.

4 is a schematic block diagram of one embodiment of a spectral sensor 10 that includes a spectral sensor array 220, an analog-to-digital converter (ADC) 240, a control engine 230, and calibration data 250. The spectral sensor 10 may be configured as a single integrated circuit, a multi-chip module, a chipset, or as contained elements included in a single integrated circuit. In one example, the spectral sensor array 220 is calibrated during the manufacturing and/or testing process of the spectral sensor array 220 to correct for variations inherent in manufacturing processes, such as variations between the mold tools of sensors (inter-die variations), as well as variance between the mold tools (intra-die variance). In one example, the calibration data is stored in a memory (such as flash memory) included in the spectral sensor 10 or in a memory included in the same integrated circuit as the spectral sensor array 220.

In one embodiment, the analog output of the spectral sensor array 220 is converted by an analog-to-digital converter (ADC) 240 for input to the control engine 230. In one example, calibration data collected during manufacture and/or testing of the spectral sensor array 220 is stored in memory as calibration data 250, and the control engine 230 uses calibration data 250 to "correct" the output received from the ADC 240. In another example, calibration data 250 may also be collected and/or heuristically modified by the implementation of the spectral sensor 10. In another example, calibration data 250 may be collected and/or modified during manufacture, testing, or use of a user device in which the spectral sensor 10 is enabled.

In another example, the spectral sensor consists of a plurality of single photon detectors, such as single photon avalanche diodes (SPADS) or other microphoton devices, configured in a spectral sensing array or pseudoarray. The digital information from the resulting spectral sensing array can then be input directly to the control engine 230. Examples of SPADS include, but are not limited to, single pixel silicon, InGaAs detectors, and two-dimensional arrays of CMOS detectors.

In the example of 4 Calibration data 250, as above with respect to 3 discussed, to correct the output of the spectral sensor array 220 over a variety of input/operating conditions. In the example, the control engine 230 provides a calibrated sensor output from the spectral sensor array 220 using calibration data 250. Calibrating the output of the spectral sensor array 220 in this manner involves mathematical correction requiring a matrix that requires comparatively complex mathematics. For example, matrix inversion can be used to construct a correction matrix for the example, however, the resulting correction matrix would likely be relatively imperfect while also being extremely complex to develop and/or implement.

5A provides a schematic block diagram of an embodiment of a spectral sensor 10, wherein hardware and/or software capable of providing artificial neural network functions, referred to as artificial neural network 370, is included in the spectral sensor 10. In one example, the artificial neural network 370 in a spectral sensor 10 is included in a module or integrated into one or more elements of the spectral sensor 10 and is configured to provide a corrected output based on a coordinated output of the artificial neural network 370. In one example, calibration data 350 may be provided directly to the artificial neural network 370. In another example, calibration data 350 may be used by the control engine 230 shown in 4 to provide a calibrated output from the sensor array 320 to the artificial neural network 370.

In another example, the artificial neural network 370 is implemented by an external computing device. In another example, the artificial neural network 370 is implemented in another physical layer or another element of a "stacked sensor" using 3D integration techniques. In one example, the artificial neural network may be massively connected in parallel to the spectral sensor using 3D stacking. For example, each pixel or filter field may be configured to include a direct connection to one or more artificial neural network nodes.

The artificial neural network 370 includes an interconnected structure of "artificial neurons" that act as a path for data transmission. A conventional computer system may consist of a number of simple, highly interconnected processing elements that process information with their dynamic state response to external inputs. In contrast, in the context of the artificial neural network 370, a neuron may produce a linear or a nonlinear response. In one example, a nonlinear artificial network is created by connecting nonlinear neurons; such a nonlinear system may include inputs that are not proportional to the outputs.

In one example, the artificial neural network 370 is “loaded” with weights or coefficients derived from a training process such as that in 8B and described in the accompanying text. The coefficients may be loaded directly into the artificial neural network 370 or stored with calibration data 350 for input to the artificial neural network 370.

5B shows a schematic block diagram of an embodiment of a spectral sensor 10 wherein an artificial neural network is provided with temperature data to improve the accuracy and/or precision of a corrected output. In one embodiment, temperature data 390 is provided to the artificial neural network 370. In one example, temperature data 390 is collected from a temperature sensing device external to the spectral sensor 10 and may be transmitted to the sensor array continuously or intermittently using a duty cycle where the temperature signal is active for only a fraction of a certain period of time. The temperature data 390 may also be provided continuously or using a duty cycle to the artificial neural network 370.

The temperature data 390 may be stored in a dedicated memory or in a memory shared with other spectral sensor elements. In one example, temperature data 390 is stored as numerical values representative of the actual temperature, such as a voltage or resistance value. In another example, the temperature data 390 is converted to a temperature value before being transmitted to the artificial neural network 370.

6A provides a simplified representation of an artificial neural network such as an artificial neural network 370 in the context of a spectral sensor such as a spectral sensor 10. In the example, a measured output response 410, such as the calibrated sensor output 260, is mapped to the 4 reference is made to a input layer 400 of the artificial neural network 470, where the input layer 400 includes artificial neurons (nodes) 450 connected to one or more nodes of the hidden layer 402 and an output layer 420 of nodes. In another example, there is no hidden layer 402, with nodes of the input layer 400 connected to nodes of the output layer without a hidden layer of nodes. In another example of a so-called single-layer neural network, the inputs to the artificial neural network 470 are fed directly to the nodes of the output layer using a series of weights, the sum of the products of the weights and the inputs in each node is calculated, and if the value is above a threshold (typically 0), the neuron is triggered and takes the activated value (typically 1); otherwise, the deactivated value (typically -1) is used.

6B provides a simplified representation of an artificial neural network, sometimes referred to as a multilayer perceptron, where each node in one layer has directed connections to the nodes of the subsequent layer, with multiple layers of computational units interconnected in a feedforward arrangement. For example, if a value within the perceptron nodes 510 (nodes) represents the explicit threshold of each node (which can be factored out so that all nodes have the same thresholds, marked with arrows representing the weight of the inputs. This network assumes that if the threshold is not met, zero (not -1) is output. Accordingly, in the example, a two-layer neural network can compute XOR.

In a simplified example, an input 500 is provided to the neural network and a required target response is set at the output 520, and an error is obtained from the difference of the desired response together with the output of the real system.

The error information is fed back to the system, and it makes many adjustments to its parameters in a systematic order. After repeating this process for a sufficiently large number of training cycles, the network can converge to a state where the error of the calculations is small and the network has "learned." One example applies a general method of nonlinear optimization, where the network calculates the derivative of the error function with respect to the network coefficients or weights and changes the coefficients so that the error decreases.

Various forms of neural networks can be used to provide spectral correction, including but not limited to feedback artificial neural networks, feedforward artificial neural networks, classification prediction artificial neural networks, and others.

7A is a logic diagram illustrating an exemplary method for training an artificial neural network to provide corrected spectra. In step 600, the spectral output from a sensor array, such as sensor array 220, is collected over a range of input power and wavelengths as shown in the table that 7A In one example, the sensor array may be presented with a sweep of monochromatic light over a desired wavelength range while the input power is fixed; the input power may then be increased in steps and swept at each desired input power setting for the desired wavelength range. In an alternative example, the input power may be increased in fixed steps for each fixed wavelength of a desired wavelength range. In another alternative example, selected wavelength and input power pairs such as those shown in the table of 7A shown, can be chosen statistically or randomly (or pseudorandomly) to fill a matrix. In another example, a matrix such as the one in 7A The matrix shown can be modified using a fractional factorial design to reduce training runs and resources.

In the example of 7A light wavelengths and input power are considered. In other examples, input factors can be varied. Example input factors can include, for example, a signal or other indication of an illumination source type (such as laser, natural light, light emitting diode (LED) light, etc.), an object type being sensed (e.g. human skin, outdoor scene, etc.), and/or an intended use for the spectral output (such as health analytics, white balance, etc.). For other input factors, see and in the accompanying text. In one example, multidimensional matrices can be used to accommodate a variety of input factors. In a related example, multidimensional matrices designed for multiple input factors can use full and fractional factorial designs.

In the example, the matrix of Figure 7A contains monochromatic light at wavelengths from 720 nm to xxx nm in 10 nm steps for each of the 1-y input power units. In practice, wavelengths could be incremented in smaller increments such as 1 nm or in increments > 10 nm, and the input power can be incremented in a similar manner.

The procedure of 7A results in a characterization model of a hypothetical spectrometer M of dimensions L x S x N, where L is the number (#) of input power increments, S is the # of wavelength increments, and N is the number of filter "patches." A filter patch in the example is the number of filter elements across the sensor array that select for the same wavelength transmission. In one example, this spectrometer "model" can provide full resolution spectra (single wavelength increments of monochromatic light) and output N values. For example, for a 64-channel spectrometer, there would be 64 outputs. The method begins in step 600, where output is collected for several pairs of input power and wavelength, and continues in step 610 where a large number of synthetic input spectra are generated. In one specific example, 100 million spectra, covering essentially all conceivable permutations the spectrometer could be subjected to. In another specific example, the synthetic input spectra can be selected for targeted permutations of the spectrum's exposure conditions. These synthetic input spectra can then be used to train a corrected spectrum, as discussed above.

In step 620, the artificial neural network is trained (i.e., the coefficients or weights of the neural network are determined) such that when synthetic spectra are provided to the spectrometer, the artificial neural network outputs the synthetic spectra (i.e., the error in the spectrometer is minimized to an acceptable amount). In step 630, the coefficients or weights are fed into the artificial neural network (such as the artificial neural network 370 of 5 ). The spectrometer can then correct any input light into the spectral array to produce a corrected spectrum.

7B is a logic diagram illustrating an example method for adaptively correcting spectral output from a sensor array. The method begins in step 640, where the sampled spectrum from an object or scene is recorded. In one example, recording may include one or more of storing the sampled spectrum in memory, holding the sampled spectrum in a buffer, and providing readings as received to a processing module. The method continues in step 642, where the recorded spectrum is analyzed. In one example, analysis may include analyzing the recorded spectrum to determine one or more of intensities at selected wavelengths, total energy, ratios for recorded peaks in the spectrum (such as peak/valley ratios or ratios between peaks), relative smoothness or relative spikiness signal, etc.).

In step 644, the method continues, determining a calibration method based on the analyzed spectrum and one or more additional input factors. Additional input factors may include, for example, a signal or other indication of a type of illumination source (such as laser, natural light, LED light-emitting diode light, etc.), a type of object being scanned (such as human skin, outdoor scene, etc.), and/or an intended use for the spectral output (such as health analytics, white balance, etc.). See 8D and the accompanying text for additional input factors. In an example, the calibration procedure may include a matrix such as that shown in 7A Other calibration methods include one or more calibrations using one or more of two wavelengths (such as two wavelengths useful for measuring peripheral oxygen saturation (SpO2)), calibrations using principal component analysis, and calibrations using other schemes. In step 644, the particular calibration method is applied to the sampled spectrum. In step 648, when the correction is complete, the corrected sampled spectrum is analyzed to determine if it is within predetermined limits, and if the corrected sampled spectrum is not within predetermined limits, step 642 may be repeated and/or another calibration may be used until the corrected sampled spectrum is within predetermined limits. In one example, the predetermined limits may be based on one or more input factors, such as one or more metadata associated with an object or scene, a classification boundary, a lookup table, a formula, or a calibration method used.

7C provides a logic diagram illustrating another exemplary method for adaptively correcting the spectral output from a sensor array. The method begins in step 650 an optical sensor array associated with the spectral sensor system that receives light at N input powers for each wavelength of light of M wavelengths of light to generate N x M wavelength and input power pairs. In one example, each wavelength of light of the M wavelengths of light and each input power of light of the N input powers of light are received by the optical sensor array in a predetermined sequence. In another example, each optical sensor of the optical sensor array is associated with an optical filter of a plurality of sets of optical filters, and each optical filter of a set of optical filters is configured to pass light in a different range of wavelengths. The method continues in step 652, wherein the spectral sensor system generates a plurality of signals to produce a generated signal for each wavelength and input power pair, each of the plurality of signals being representative of the received light at each wavelength and input power pair of the N x M wavelength and input power pairs. In step 654, the spectral sensor system determines a plurality of error coefficients, each error coefficient of the plurality of error coefficients representative of the difference between a generated signal and a target value for a wavelength and input power pair. In step 656, the sensor system uses the plurality of error coefficients to train an artificial neural network configured to adjust a spectral response generated by the optical sensor array, and in step 658, the output of the trained artificial neural network may be used to adjust the spectral response generated by another optical sensor array.

In a specific implementation example, this can be 7C called an artificial neural network may be configured to receive information associated with a scene sensed by a different spectral sensor. In another example, the output of the trained artificial neural network is provided to another artificial neural network associated with a different optical sensor array (in a different spectral sensor system), the other artificial neural network configured to receive information associated with a scene sensed by the different spectral sensor system.

In an example, this can be 7C The artificial neural network mentioned may be trained with coefficients for N × M wavelength and input power pairs together with corrections based on other input factors, such as the input factors referred to with reference to 7A In another example, a trained artificial neural network may be used to provide weighting for one or more artificial neural networks associated with other spectral sensor systems, which may themselves be trained based on various input factors.

8A shows an exemplary spectral sensor, where input parameters 310, such as a sampling of monochromatic light over selected wavelengths at selected input power units, are provided to the spectral sensor 10 and an uncorrected output response 500 is collected (such as L × S × N from step 600 of 7A) . In one example, a spectral sensor 10 includes calibration data collected during testing and/or manufacturing of the spectral sensor 10, where the calibration data is specific to the spectral array 220 of the spectral sensor 10. In one example, the calibration data is stored in a flash memory associated with the spectral sensor 10.

In the example, the uncorrected output response 500 is a measure of how the spectral sensor 10 behaves when swept with monochromatic light at incremental input power levels. In an "ideal" spectral sensor, each sensor or pixel in combination with an associated filter element produces a signal that is a substantially perfect representation of the input power and light provided to the spectral sensor 10, such that a linear sequence of wavelengths at a given input power results in a spectral output response that substantially matches the input power and light provided to the spectral sensor 10 (i.e., the spectral output response would be "ideal"). Since no spectral sensor operates in such an ideal manner, the uncorrected output response 500 deviates from the ideal spectral output response and requires correction to accurately represent the input parameters 310.

In 8B An example spectral sensor using an artificial neural network is shown. In one example, raw spectral data, such as the uncorrected output response 500 from 8A , used to create a spectrometer model 510 for the spectral sensor 10, such that the spectrometer model 510 reflects the non-ideality of the spectral sensor 10 for all input parameters 310 provided to the spectral sensor 10, as in the example of 8A .

In the example, artificial spectra 560 are received at spectrometer model 510, the output of which is then input to an artificial neural network such as artificial neural network 370 and compared to a desired or expected spectral output for artificial spectra 560. In one example, error 530 is repeatedly determined for the artificial spectra 560 and input to artificial neural network 370 to generate coefficients that reflect the difference between the artificial spectra 560 and spectrometer model 510. The generated coefficients may then be stored and subsequently loaded into a generic artificial neural network to correct the spectrum in an associated spectral sensor. In one example, a number of artificial spectra 560 is determined based on the desired precision and accuracy for spectral sensor 10 and may, for example, exceed 100 million spectra.

In one embodiment, the spectral sensor 10 may filter patterns over the spatial regions of a sensor array (such as the spectral sensor array 100 of 1 ) to optimize a corrected spectral output response 520. For example, the filter pattern can be applied across the spectral sensor array (such as the spectral sensor array 100 of 1 ) and/or repeated multiple times across the spectral array. For example, the repeated patterns may reduce shading effects and other effects due to non-uniformities that may be the result of other elements associated with the spectral sensor 10, such as collimation optics.

In another embodiment, the effect of changes in lighting and/or tilt of the scene may be corrected for and/or modified by the artificial neural network 370. Scene tilt may be described as the relationship between a scene or object on a vertical and/or horizontal axis relative to an imager, such as a spectral imager. For example, a scene that is perpendicular to the line of sight of an imaging system (such as a camera) in a vertical and horizontal axis could be described as having a tilt of zero degrees (0 degrees). If the line of sight is described as a line extending from the viewpoint of an imager to a scanned object or area (e.g., a stage), then a tilt of 20 degrees could indicate that the scanned object deviates 20 degrees from perpendicular to the line of sight.

In one example, synthetic spectra, such as artificial spectra 560, include measured parameters for various tilt and illumination increments that can be used in training to minimize the effects of those parameters. In one example, correcting for tilt and/or illumination can reduce the requirement to collimate elements in the sensor array. In a specific example, measured parameters such as a scene tilt or illumination change could be used with an artificial neural network 370 configured to allow a user to impose an effect on an image, such as a scene tilt. In one example, a tilt shift effect could be provided so that an image with 0 degrees of tilt could reflect a variable amount of tilt to provide a tilt shift effect. In a specific related example, measured scene tilt could be used with an artificial neural network 370 to allow a user to manipulate the focal plane of an image to correct for distortion due to tilt. In another specific related example, a measured scene tilt could be used with an artificial neural network 370 to allow a user to manipulate the focal plane of an image to soften the convergence of vertical and/or horizontal lines of another feature in the image.

In another embodiment, the effect of changes in ambient light and/or temperature changes may be corrected by the artificial neural network 370. In one example, the synthetic spectra, such as the artificial spectra 560, include measured parameters for various ambient light conditions and temperatures that may be used in training to minimize the effects of these parameters.

In a specific example of implementation and operation, a spectral sensor system comprises a plurality of optical sensors arranged on an integrated circuit, wherein the plurality of optical sensors are arranged in an array and an interface between the plurality of optical sensors and a first processing device configured to communicate information between them. In an example, a plurality of sets of optical filters are configured as a layer located on the plurality of optical sensors, wherein a set of optical filters of the plurality of sets of optical filters comprises a plurality of optical filters, and each optical filter of the plurality of optical filters to pass light in a different wavelength range. Included in the example is a memory configured to interface and communicate with the first processing device and store calibration data associated with the plurality of sets of optical sensors, wherein a second processing device comprises an artificial neural network configured to correct a spectral response generated by the plurality of optical sensors, along with an interface between the first processing device and the second processing device.

8C shows an example spectral sensor in which a physical scene 160 is provided to a spectral sensor 10, which is then output to a trained artificial neural network 370 to provide a corrected spectral output response 520 for the physical scene 560. 8D shows an example spectral sensor where physical scene metadata is provided to the artificial neural network, such as artificial neural network 370. In one example, physical scene metadata 570 is predetermined for use by artificial neural network 370. In one example, if a particular plant is the subject of analysis, known spectral responses for that plant may be used to increase the efficiency of artificial neural network 370. This could be accomplished by optimizing one or more of the measurement regions, increasing the granularity of measurements in a portion of the physical scene, or otherwise manipulating the analysis based on metadata 570. Physical scene metadata 570 may include data provided by additional sensors, such as, but not limited to, gyroscopes, magnetometers, accelerometers, temperature sensors, and image sensors.

In a specific example of implementation and operation, the metadata 570 may include information about a physical scene 560 that the artificial neural network 370 either cannot correct or would produce a corrected spectral output response 520 outside of acceptable limits. In one example, the artificial neural network 370 may be configured to provide a notification, such as a signal, indicating that a correction is not within acceptable limits. In another example, the artificial neural network 370 may be configured to provide a notification, such as a signal, indicating that a correction is within acceptable limits, effectively informing a user that the corrected spectral output response 520 can be trusted. Examples of metadata 570 information that the artificial neural network 370 either cannot correct or would be outside acceptable limits include one or more of too little input light, sensor saturation, or the angle of incidence for incoming light to be corrected by the artificial neural network 370.

In another specific example of implementation and operation, the artificial neural network 370 may be configured with a variety of operating modes. In one example, the artificial neural network 370 may be configured with a high accuracy, higher performance mode that produces a relatively high quality spectral output but has a relatively higher power consumption, or the artificial neural network 370 may be configured with a lower accuracy, lower performance mode where fewer calculations are performed to reduce power consumption on a relative basis. In another specific example of implementation and operation, a reduced complexity operation may be configured to correct one or more scenarios. For example, the artificial neural network 370 may be configured to select a condition or set of conditions, such as a mean light intensity, and angle of incidence for incident light collimated at an angle of 6 degrees, before the correction is performed by the artificial neural network 370. In one example, the reduced set of conditions may provide, for example, reduced power consumption. In a related example, a specific correction may be provided by an artificial neural network 370 for use in a non-neural-based processor to reduce the relative performance or cost of spectral correction.

In another specific example of implementation and operation, the artificial neural network 370 may be configured to provide a confidence or accuracy value with a corrected spectral output response 520. For example, if a red spot (or red tile) is measured at 50% input power, the resulting corrected spectral output response 520 for the measurement includes an absolute/relative accuracy of 1% at the wavelength of 700 nm and an absolute/relative accuracy of 5% at 500 nm. For example, if a red spot (or red tile) is measured at 50% input power, the resulting corrected spectral output response 520 for the measurement has an absolute/relative accuracy of 1% at the wavelength of 700 nm and an absolute/relative accuracy of 5% at 500 nm. In an example, if the corrected spectral output response 520 has a relatively low confidence value between 500 nm and 550 nm to be accurate, the subsequent application layer could assign less importance to that portion of the corrected spectrum.

In one example, a neural network correction algorithm may include one or more weighted loss functions (sometimes referred to as weighted cost functions) associated with a region and/or area of interest, such as a portion of the physical scene 560 or one or more identified objects or areas from the physical scene 560. In one example, a weighted loss function may be used to increase the sensitivity of a spectral sensing system, such as spectral sensor 10, for a particular region and/or area being sampled. In a specific example of implementation and operation, one or more weighted loss functions may enable the hardware of the spectral sensor system (such as spectral sensor 10) to be adaptable for use in areas with different requirements and performance expectations. In one example, the one or more weighted loss functions are fixed at manufacture. In another example, the one or more weighted loss functions may be manually selected. In another example, the one or more weighted loss functions are determined based on an algorithm, such as a pattern recognition algorithm.

In a specific related example, the measurement of hydration, such as the level of hydration in the skin, may be implemented using data from spatially separated regions of a sensor to increase the accuracy and/or precision at 5 specific portions of the spectrum used for hydration analysis. In the example, the accuracy and/or precision of the spectrum may be relatively less relevant in other spectral regions, thus the spectral sensor system may reduce or eliminate samples from those spectral regions.

In another example, the spectral sensor 10 may be used in combination with the artificial neural network 370 to generate a confidence image, where the confidence image provides information sufficient to determine whether the actual measured spectrum is reasonable. For example, the artificial neural network 370 may provide an indication of the deviation of the measured spectrum from the already determined training data. The deviation may then be used to indicate abnormal spectra (spectra that cannot normally occur) that indicate an incorrect measurement.

In another example, the spectral sensor in combination with the artificial neural network 370 may be configured to provide a correction of the spectral response as an intermediate step before outputting a corrected spectrum. In the example, an artificial neural network corrected spectral response is passed to the subsequent layer and an algorithm or the artificial neural network is used to determine a property or properties derived from the measured spectrum. For example, a property such as the water content and/or the oxygen content in a medium to be analyzed may be determined, the determination being based on the corrected spectral response provided by the artificial neural network without the need for a corrected spectrum.

In one embodiment, physical scene metadata 570 and/or other data input to artificial neural network 370 is provided in substantially real-time (live) so that artificial neural network 370 can continually revise the corrected output. In another embodiment, physical scene metadata 570 and/or other data is previously stored and provided to artificial neural network 370 as required and/or feasible.

It is noted that terminologies as may be used herein, such as bit stream, stream, signal sequence, etc. (or their equivalents), have been used interchangeably to describe digital information whose content corresponds to any of a number of desired types (e.g., data, video, voice, text, graphics, audio, etc., each of which may be referred to generically as "data").

As used herein, the terms "substantially" and "approximately" provide an industry-accepted tolerance for the corresponding term and/or the relativity between elements. For some industries, an industry-accepted tolerance is less than one percent and for other industries, the industry-accepted tolerance is 10 percent or more. Other examples of industry-accepted tolerance include chen from less than one percent to fifty percent. Industry accepted tolerances correspond to, but are not limited to, component values, integrated circuit process variations, temperature variations, rise and fall times, thermal noise, dimensions, signaling errors, discarded packets, temperatures, pressures, material compositions, and/or performance metrics. Within an industry, tolerance variations of accepted tolerances can be more or less than a percentage level (e.g. dimensional tolerance of less than +/- 1%). Some relativity between elements can range from a difference of less than a percentage to a few percent. Other relativity between elements can range from a difference of a few percent to the magnitude of the differences.

As may also be used herein, the terms “configured to,” “operably coupled to,” “coupled to,” and/or “coupling” include direct coupling between elements and/or indirect coupling between elements via an intervening element (e.g., an article includes, but is not limited to, a component, element, circuit, and/or module), for example, for indirect coupling, the intervening element does not change the information of a signal, but can adjust its current level, voltage level, and/or power level. As may further be used herein, inferred coupling (i.e., when an element is coupled to another element by inference) includes direct and indirect coupling between two elements in the same manner as “coupled to.”

As may be further used herein, the term "configurable to," "operable to," "coupled to," or "operably coupled to" indicates that an item includes one or more of power ports, input(s), output(s), etc., to perform one or more of its respective functions when activated, and may further include a derivative coupling to one or more other items. As may be further used herein, the term "linked to" includes the direct and/or indirect coupling of separate items and/or an item embedded within another item.

As may be used herein, the term "compare favorably" indicates that a comparison between two or more elements, signals, etc., provides a desired relationship. For example, if the desired relationship is that signal 1 has a larger magnitude than signal 2, a favorable comparison may be achieved if the magnitude of signal 1 is larger than that of signal 2 or if the magnitude of signal 2 is smaller than that of signal 1. As may be used herein, the term "compare unfavorably" indicates that a comparison between two or more elements, signals, etc. does not provide the desired relationship.

As used herein, one or more claims may contain the phrase "at least one of a, b, and c" in a specific form of this generic form, or of this generic form "at least one of a, b, or c" with more or fewer elements than "a," "b," and "c." In either formulation, the phrases are to be interpreted identically. In particular, "at least one of a, b, and c" is synonymous with "at least one of a, b, or c" and means a, b, and/or c. As an example, it means: "a" only, "b" only, "c" only, "a" and "b," "a" and "c," "b" and "c," and/or "a," "b," and "c."

As may also be used herein, the terms "processing module,""processingcircuit,""processor,""processingcircuit," and/or "processing unit" may refer to a single processing device or multiple processing devices. Such a processing device may be a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, and/or any device that manipulates signals (analog and/or digital) based on hard coding of the circuit and/or operating instructions. The processing module, module, processing circuit, processing circuit, and/or processing unit may be a memory and/or an integrated storage element, or may further comprise a memory and/or an integrated storage element that may be a single storage device, multiple storage devices, and/or embedded circuitry of another processing module, module, processing circuit, processing circuit, and/or processing unit. Such a storage device may be a read-only memory, a random access memory, a volatile memory, a non-volatile memory, a static memory, a dynamic memory, a flash memory, a cache memory, and/or any device that stores digital information. It should be noted that if the processing module, module, processing circuit, processing circuit, and/or processing unit includes more than one processing device, the processing processing devices may be centrally located (e.g. directly interconnected via a wired and/or wireless bus structure) or distributed (e.g. cloud computing via indirect coupling via a local area network and/or a wide area network). Further, it should be noted that if the processing module, module, processing circuit, processing circuit and/or processing unit implements one or more of its functions via a state machine, an analog circuit, a digital circuit and/or a logic circuit, the memory and/or the storage element in which the corresponding operating instructions are stored may be embedded in or outside of the circuit comprising the state machine, the analog circuit, the digital circuit and/or the logic circuit. It should be further noted that the storage element may store and the processing module, module, processing circuit, processing circuit and/or processing unit executes hard-coded and/or operating instructions that correspond to at least some of the illustrated steps and/or functions in one or more of the figures. Such a storage device or element may be included in an article of manufacture.

One or more embodiments have been described above using method steps that illustrate the performance of specified functions and relationships thereof. The boundaries and sequence of these functional building blocks and method steps have been arbitrarily defined herein for ease of description. Alternative boundaries and sequences may be defined so long as the specified functions and relationships are properly performed. Further, the boundaries of these functional building blocks have been arbitrarily defined for ease of description. Alternative boundaries could be defined so long as the particular significant functions are properly performed. Similarly, flowchart blocks may also have been arbitrarily defined herein to illustrate particular significant functions.

To the extent utilized, the block boundaries and sequence of the flowchart could have been defined differently and still perform the particular significant functionality. One of ordinary skill in the art will also recognize that the functional building blocks and other illustrative blocks, modules, and components herein may be implemented as shown or by discrete components, application specific integrated circuits, processors executing appropriate software, and the like, or any combination thereof.

Additionally, a flowchart may include a "start" and/or "next" indicator. The "start" and "next" indicators indicate that the steps depicted may optionally be incorporated into or otherwise used in conjunction with one or more other routines. Additionally, a flowchart may include a "finish" and/or "next" indicator. The "finish" and/or "next" indicators reflect that the steps depicted terminate as described and shown, or may optionally be incorporated into or otherwise used in conjunction with one or more other routines. In this context, "start" indicates the beginning of the first step presented and may be preceded by other activities not specifically shown. Furthermore, the "next" indicator reflects that the steps depicted may be performed multiple times and/or followed by other activities not specifically shown. While a flowchart indicates a particular order of steps, other orderings are also possible, provided the principles of causality are maintained.

The one or more embodiments are used herein to illustrate one or more aspects, one or more features, one or more concepts, and/or one or more examples. A physical embodiment of an apparatus, article of manufacture, machine, and/or method may include one or more of the aspects, features, concepts, examples, etc. described with reference to one or more of the embodiments discussed herein. Further, the embodiments may include from figure to figure the same or similarly named functions, steps, modules, etc., which may use the same or different reference numbers, and as such, the functions, steps, modules, etc. may be the same or similar functions, steps, modules, etc., or different.

Unless expressly stated otherwise, signals to, from, and/or between elements in a figure of any of the figures shown herein may be analog or digital, continuous time or discrete time, and single-ended or differential. For example, when a signal path is shown as a single-ended path, it also represents a differential signal path. Similarly, when a signal path is shown as a differential path, it also represents a single-ended signal path. While one or more particular architectures are described herein, other architectures may also be implemented that include one or more data buses not expressly shown, direct connectivity between elements and/or indirect coupling between other elements, as would be recognized by one of ordinary skill in the art.

The term "module" is used in the description of one or more of the embodiments. A module implements one or more functions via a device such as a processor or other processing device or other hardware that may include or operate in conjunction with a memory in which operating instructions are stored. A module may operate independently and/or in conjunction with software and/or firmware. As also used herein, a module may include one or more submodules, each of which may be one or more modules.

As may be further used herein, a computer-readable storage includes one or more storage elements. A storage element may be a separate storage device, multiple storage devices, or a set of storage locations within a storage device. Such a storage device may be read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and/or any device that stores digital information. The storage device may be in the form of solid-state memory, hard disk storage, cloud storage, USB flash drive, server storage, computer memory, and/or other physical medium for storing digital information.

While certain combinations of various functions and features of the one or more embodiments have been expressly described herein, other combinations of these features and functions are also possible. The present disclosure is not limited to the particular examples disclosed herein and expressly includes these other combinations.

Claims (20)

A spectral sensor system comprising: a plurality of optical sensors (100) arranged on an integrated circuit and forming a spectral sensor (10), the plurality of optical sensors (100) arranged in an array (170); an interface between the plurality of optical sensors (100) and a first processing device configured to transmit information therebetween; a plurality of sets of optical filters (110, 120, 130, 160) configured as a layer disposed on the plurality of optical sensors (100), wherein a set of optical filters (110, 120, 130, 160) of the plurality of sets of optical filters (110, 120, 130, 160) comprises a plurality of optical filters (110, 120, 130, 160), each optical filter (110, 120, 130, 160) of the plurality of optical filters (110, 120, 130, 160) being configured to transmit light (180) in a different wavelength range; a memory configured to interface and communicate with the first processing device, the memory configured to store calibration data (250, 350) associated with the plurality of sets of optical sensors (100); a spectrometer model (510) reflecting the non-ideality of the spectral sensor (10); a second processing device comprising an artificial neural network (370) configured to correct a spectral response generated by the plurality of optical sensors (100); based on an error (530) determined for artificial spectra (560) and input to the artificial neural network (370) to generate coefficients as weights of the artificial neural network (370) reflecting the difference between the artificial spectra (560) and the spectrometer model (510); and an interface between the first processing device and the second processing device configured to transfer information therebetween. Spectral sensor system according to Claim 1 , wherein the plurality of optical filters (110, 120, 130, 160) of each set of the plurality of sets of optical filters (110, 120, 130, 160) are arranged in a pattern, the pattern being repeated for at least some of the sets of the plurality of sets of optical filters (110, 120, 130, 160) across the array (170). Spectral sensor system according to Claim 1 or Claim 2 wherein the optical filters (110, 120, 130, 160) in at least some of the sets of the plurality of sets of optical filters (110, 120, 130, 160) are interference filters. Spectral sensor system according to Claim 1 or Claim 3 , where the interference filters are Fabry-Perot filters. Spectral sensor system according to Claim 1 wherein the second processing device is configured to receive information associated with a scene sensed by the spectral sensor system. Spectral sensor system according to Claim 5 , where the information associated with the scene includes metadata associated with the scene. Spectral sensor system according to Claim 5 or Claim 6 , wherein the information associated with the scene comprises at least one of the following: 1) a temperature; 2) a measure of ambient light; 3) a measure of illumination of the scene; 4) a slope for the scene, where the slope is a deviation of the scene on at least one vertical or horizontal axis from vertical relative to a line of sight of the spectral sensor system; 5) an orientation of the spectral sensor system; 6) an orientation of the scene; 7) an angular velocity of the spectral sensor system; 8) an angular velocity of one or more objects in the scene; 9) a magnetic field associated with the scene; 10) a magnetic field associated with the spectral sensor system; and 11) a magnetic dipole moment of the spectral sensor system. Spectral sensor system according to Claim 5 or Claim 6 , wherein the information associated with the scene is provided by a user, wherein the user-provided information includes information sufficient to identify an object in the scene, wherein the second processing device is configured to correct the spectral response generated by the plurality of optical sensors (100) based at least in part on the object. Spectral sensor system according to Claim 1 or Claim 5 , further comprising: a user notification mechanism, wherein the second processing device is configured to provide a representation of a corrected spectral response to a user using the user notification mechanism. A method for performance by one or more devices of a spectral sensor system, the method comprising: receiving light (180) having N input powers for each wavelength of light of M wavelengths of light (180) by an optical sensor array (170) associated with the spectral sensor system to generate N x M wavelength and input power pairs, each wavelength of light (180) of the M wavelengths of light (180) and each input power of light (180) of the N input powers of light (180) being received by the optical sensor array (170) in a predetermined sequence, each optical sensor of the optical sensor array (170) being associated with an optical filter (110, 120, 130, 160) of a plurality of sets of optical filters (110, 120, 130, 160), each optical filter (110, 120, 130, 160) of a set of optical filters (110, 120, 130, 160) configured to pass light (180) in a different wavelength range; generating a plurality of signals, each signal of the plurality of signals representative of the received light (180) at each wavelength and input power pair of the N x M wavelength and input power pair to generate a generated signal for each wavelength and input power pair; determining a plurality of error coefficients, each error coefficient of the plurality of error coefficients representative of a difference between a generated signal and a target value for a wavelength and input power pair; Using the plurality of error coefficients to train an artificial neural network (370), the artificial neural network (370) configured to adjust a spectral response generated by the optical sensor array (170) to generate a corrected spectrum. Procedure according to Claim 10 wherein the optical filters (110, 120, 130, 160) in each set of the plurality of sets of optical filters (110, 120, 130, 160) are arranged in a pattern, the pattern being repeated for at least some of the sets of the plurality of sets of optical filters (110, 120, 130, 160) across the array (170). Procedure according to Claim 10 or Claim 11 wherein the optical filters (110, 120, 130, 160) in at least some of the sets of the plurality of sets of optical filters (110, 120, 130, 160) are interference filters. Procedure according to Claim 12 , where the interference filters (110, 120, 130, 160) are Fabry-Pérot filters. Procedure according to Claim 10 or Claim 12 wherein the artificial neural network (370) is configured to receive information associated with a scene sensed by another spectral sensor (10). Procedure according to Claim 14 , where the information associated with the scene includes metadata associated with the scene. Procedure according to Claim 14 or Claim 15 , wherein the information associated with the scene comprises at least one of the following properties: 1) a temperature; 2) a measure of ambient light; 3) a measure of illumination of the scene; 4) a slope for the scene, where the slope is a deviation of the scene on at least one vertical or horizontal axis from vertical relative to a line of sight of the spectral sensor system; 5) an orientation of the spectral sensor system; 6) an orientation of the scene; 7) an angular velocity of the spectral sensor system; 8) an angular velocity of one or more objects in the scene; 9) a magnetic field associated with the scene; 10) a magnetic field associated with the spectral sensor system; and 11) a magnetic dipole moment of the spectral sensor system. Procedure according to Claim 14 or Claim 15 , wherein the information associated with the scene is provided by a user, wherein the user-provided information includes information sufficient to identify an object in the scene, wherein the artificial neural network (370) is configured to correct the spectral response generated by the optical sensor array (170) based at least in part on the object. Procedure according to Claim 10 , further comprising: transmitting the plurality of signals to a first processor associated with the spectral sensor; calibrating the plurality of signals by the first processor to produce the generated signals, wherein the calibrating of the plurality of signals is based on calibration data (250, 350), the calibration data (250, 350) is based on measured variations associated with the manufacture of the optical sensor array (170). Procedure according to Claim 18 , wherein the artificial neural network (370) is associated with a second processor, the first processor configured to transmit the generated signals to the second processor. Procedure according to Claim 19 , further comprising: a user notification mechanism, wherein the second processor is configured to provide a representation of a corrected spectral response to a user using the user notification mechanism.

Images