JP6517788B2 - System and method for adaptive histopathology image decomposition - Google Patents

Description

  [001] The present invention relates to the unmixing of images such as histopathological images in fluorescence microscopy or bright field microscopy. More particularly, the present invention relates to dynamically and adaptively refining reference spectra for multispectral images or reference colors for brightfield images.

  [002] In multiple fluorescence slides of tissue specimens, different nuclei and tissue structures are simultaneously stained with specific fluorescent dyes, organic fluorescent dyes, or fluorescent counterstains, each 400 nm while emitting fluorescence in different spectral bands Overall overlap in the spectral range of ̃850 nm. More recently, quantum dots are widely used in immunofluorescent staining for biomarkers of interest because of their strong and stable fluorescence. On a typical multiplex slide, nuclear markers such as 4 ', 6-diamidino-2-phenylindole (DAPI: 4', 6- diamidono-2- phenylindole) fluorescent stain (i.e. blue stain) are Used with dots. However, other nuclear counterstains may be used, such as other fluorescent counterstains. The slide is then imaged using a multispectral imaging system (eg, a fluorescence microscope system connected to a camera or scanner or a whole slide scanner). Each channel of the imaging system corresponds to a spectral narrow band filter. Thus, the multispectral image stacks generated by the imaging system are a mixture of underlying biomarker expressions, which may sometimes be co-localized.

  [003] In bright field degradation, nuclear and histology are stained with hematoxylin and eosin (H & E: hematoxylin and eosin) or IHC staining. The slides are then scanned with a bright field scanner equipped with a CCD color camera and finally an RGB image is acquired. Similar to multispectral image analysis, reference colors for RGB images are obtained from scanned single stain bright-field images.

  [004] Taking multispectral image decomposition as an example, spectral decomposition is used to identify each pixel of the multispectral image in order to identify the individual component fluorescent dyes for the biomarker and the rate at which they appear in the mixture. , Decompose into a collection of component spectral end members (or reference spectra) and the proportion of intensity contribution to the multispectral image from each of them. The reference spectrum for the stain corresponds to the emission spectrum signature for a particular stain (eg, a fluorescent dye) when the stain is illuminated with spectra such as light of various excitation wavelengths. The amount of end member contribution, also called abundance, corresponds to the pixel values of the decomposed image, for example the gray scale pixel values of the DAPI or quantum dot decomposed image.

  [005] Accurate spectral resolution of fluorescence images is of clinical importance as it is one of the major steps in multi-pathology tissue image analysis. For example, several techniques for spectral decomposition have been proposed in the field of remote sensing. One common approach is to solve for abundance given a reference spectrum in a non-negative least square (NNLS) framework. In this case, accurate knowledge of the reference spectrum is required to accurately estimate the end member contribution. Unlike applications in the area of remote sensing where different objects such as grass and rocks can be easily distinguished from the scene, biomarkers are often co-localized in histopathological fluorescence images, so the end member spectra are extracted from the images It becomes difficult to solve for the quantity of fish. Narrow band reference spectra for quantum dots or nanocrystal biomarkers can be measured accurately from single-stained control slides, but broadband signals such as DAPI and tissue auto-fluorescence (AF) It is slide-specific and variations can be seen between images and slide specimens. Because the broadband spectrum overlaps with the narrowband spectrum, it becomes even more difficult to accurately estimate the quantum dot abundance (i.e., the resolved image). In addition, part of the DAPI spectrum may be mispredicted as AF, because the spectral signatures of DAPI and AF are similar to each other, which may lead to false estimation of the nuclear component. is there.

  [006] Non-negative matrix factorization (NMF) is widely used for blind spectral decomposition to simultaneously estimate the reference spectrum as well as the end member contribution. However, due to the non-linearity of the problem, this method is not guaranteed to converge to physically relevant and relevant solutions. This is particularly relevant to DAPI estimation, because the NMF can be degraded at each iteration, for the same reason that DAPI spectra can be confused with other similar reference spectra such as AF. is there. NMF automatically solves for the reference spectral matrix, but the algorithm can not identify which reference spectrum corresponds to DAPI and which corresponds to AF. The further framework proposed based on the end member orthogonality hypothesis (ie the assumption that the DAPI reference spectrum is orthogonal to that of the quantum dot) does not produce meaningful results for real data. Accurate DAPI degradation is very clinically important as DAPI is the most common nuclear stain. Moreover, nuclear detection serves as the first step in digital pathology image analysis, and further analysis tasks are based on reliable identification of cell nuclei. Thus, accurate decomposition results are needed. In the case of bright field imaging, hematoxylin plays an equally important role as DAPI in multispectral imaging. Thus, the exact degradation of hematoxylin is also very important.

  [007] The present disclosure presents systems and methods for adaptively optimizing a reference spectrum for a multispectral image, or a reference color for a bright field image. Here, the update of the broadband spectrum of the multispectral image is an example to explain the process, but the same procedure can be applied to the update of the reference color of the bright field image. This adaptive optimization is based on the structure detected in the decomposed broadband channel of the image. The embodiments disclosed herein detect and segment structures such as nuclei from resolved images of broadband channels, tissue structures, and their boundaries, and inferred from the identified structures and regions surrounding them Updating the reference matrix with the one or more wideband signals. Structural confidence levels can be determined based on structural shape and grayscale intensity information as well as biological criteria, using a broad band signature from only the reliable structures Update the reference spectrum in the reference matrix. An updated reference spectrum is estimated from the reliable structure region by obtaining pixel values from the original multispectral image and using it to replace the existing reference spectrum in the reference matrix. Subsequent decomposition operations with the updated reference matrix yield improved results. The reference spectrum update and the subsequent decomposition can be repeated iteratively until an optimized reference matrix is obtained. The optimization may be based on reaching a threshold number of iterations, or on convergence or stability of the reference matrix, or any combination thereof. The optimized reference matrix leads to a more reliable decomposition of hyperspectral data than in the case of existing methods. The decomposition operation can be performed iteratively on the same or different areas of the image, or the entire image. Custom regions can be defined, which allow analysis of images with various broadband signatures. Different operations can be performed in parallel for different regions, which allows for the efficient processing of multiple multiplexed fluorescent slides. In one exemplary embodiment of the invention, a non-transitory computer readable medium for storing computer executable instructions to be executed by a processor to perform an operation is disclosed, which comprises an initial reference spectrum and a color. Decomposing the image or region of the image comprising the mixture of signals using the reference matrix, estimating the updated reference spectrum or color with respect to the image or the region of the image, and updating the reference spectrum or reference color Updating the reference matrix, wherein the updated reference spectrum or reference color is used in a subsequent separation operation. In an exemplary embodiment of the invention, the resolved channel is one or more of a 4 ′, 6-diamidino-2-phenylindole (DAPI) signal, or an autofluorescent signal, and a hematoxylin (HTX) signal. It can be done.

  [008] In an embodiment of the present invention, a system for image resolution comprises a processor and a memory connected to the processor, the memory being capable of resolving a reference spectrum for a stain corresponding to the stain Estimating and decomposing an image from high-ranked subsets of multiple structures detected from different signals and decomposing the image using the reference spectrum provides an optimal reference spectrum It is for storing computer-executable instructions that, when executed by a processor, cause the processor to perform operations repeated up to.

  [009] In another embodiment of the present invention, the steps of decomposing an image containing a mixture of signals, detecting multiple structures in the decomposed signal, and ranking the structures in order of reliability Estimating a reference spectrum for the stain from a subset of structures in the plurality of structures having a degree of confidence higher than a threshold confidence, storing the reference spectrum in a reference matrix associated with the image And subsequently resolving the image using the reference matrix.

  [0010] In another embodiment of the present invention, a non-transitory computer readable medium for storing computer executable instructions is disclosed that includes a mixture of signals utilizing a reference matrix that includes an initial reference vector. Resolving the first region of the image to produce a resolved first region of the image, estimating an updated reference spectrum for the resolved first region, and for the resolved first region Processing by the processor to perform operations including generating an updated reference vector and updating the reference matrix with the updated reference vector, wherein the updated reference vector is used in a subsequent decomposition operation To be executed.

  [0011] In yet another embodiment of the present invention, a system for resolving an image is disclosed comprising a processor and a memory connected to the processor, the memory including a reference vector for the stain. Estimating and decomposing the image from the high-ranked subset of structures detected from the decomposed signals corresponding to the staining agent, and decomposing the image using the reference spectrum Store computer-executable instructions that, when executed by the processor, cause the processor to perform operations that are repeated until an optimal reference spectrum is obtained.

  [0012] In yet another embodiment of the present invention, a method for decomposing, decomposing an image containing a mixture of signals to generate a deconstructed signal, and a plurality of deconstructed signals. Staining, as indicated in the image, from the steps of detecting structures, ranking the structures in order of confidence, and a subset of the structures in the plurality of structures having a confidence higher than the threshold confidence. A method is disclosed comprising estimating a reference vector for the agent, storing the reference vector in a reference matrix associated with the image, and subsequently resolving the image using the reference matrix.

[0013] FIG. 1 is a diagram of a system for adaptive spectral decomposition in accordance with an exemplary embodiment of the present subject disclosure. [0014] FIG. 7 is an illustration of a method for adaptive spectral decomposition in accordance with an exemplary embodiment of the present subject disclosure. [0015] FIG. 3A is a diagram of an image segmented into regions for adaptive spectral decomposition, according to an exemplary embodiment of the present subject disclosure. FIG. 3B is a diagram of an image segmented into regions for adaptive spectral decomposition in accordance with an exemplary embodiment of the present subject disclosure. [0016] FIG. 6 is an illustration of a method for detecting a structure in a wideband signal of an image, according to an exemplary embodiment of the present subject disclosure. [0017] FIG. 7 is an illustration of a method for determining the confidence of a detected structure, according to an exemplary embodiment of the present subject disclosure. [0018] FIG. 7 is a diagram of ranking of nuclei using an elliptical fit, according to an exemplary embodiment of the present subject disclosure. [0019] FIG. 7 is a diagram of a median reference spectrum estimated from a plurality of high rank structures, in accordance with an exemplary embodiment of the present subject disclosure. [0020] FIG. 7 is a comparison of nuclear detection results using an initial reference spectrum to an updated reference spectrum, according to an exemplary embodiment of the present subject disclosure.

  [0021] The disclosure of the present subject matter presents systems and methods for adaptively optimizing a broadband reference spectrum for an image comprising multiple fluorescence channels, or updating a reference color for a bright field image . For each region of the image, a reference vector, ie, a reference spectrum for a fluorescence image, or a reference color for a bright field image spectrum, is updated and the region is resolved until convergence. As an application, the present disclosure presents details of a reference spectral refinement method for multispectral images, but the same technique can be applied to brightfield images.

  [0022] Taking the fluorescence image as an example, this adaptive optimization is based on the structure detected in the resolved broadband channel of the image. For example, a slide holding sample material can be scanned using a scanner connected to a fluorescence or bright field microscope system to generate a scanned image. The image is stored on a computer readable medium and comprises a mixture of several fluorescence or color channels, including one or more broadband channels such as DAPI channels. The exemplary embodiments disclosed herein detect nuclear and tissue regions from broadband channels in the resolved image and refine the reference spectrum with the broadband signatures of the biomarkers in these regions of the image. The component fluorescence channels can be separated using linear spectral decomposition processes such as the non-negative least squares method presented herein. The image may comprise a mixture of end member spectra comprising DAPI and AF, and one or more quantum dots, as described above, which are first decomposed using an initial reference matrix. The initial reference spectrum of each marker can be taken from control data, control image, or can be estimated from the image under observation. For example, if an initial broadband reference signal or initial reference spectrum for a stain observes the ubiquitous distribution of the stain's unique signature over the entire area of the image or image, and is illuminated with a known signature associated with the stain This can be determined by identifying the signal based on a comparison of the signature spectrum emissions / outputs of the stains. The initial reference spectra for a particular stain can be retrieved from a database containing one or more reference matrices. The initial reference spectrum or reference color may correspond to the reference matrix used in the resolving operation.

  [0023] Embodiments disclosed herein detect and segment resolved broadband signals and structures such as nuclei from the resolved broadband signal channels, tissue structures, and boundaries thereof. And performing an operation including estimating the updated reference spectrum from the detected structure. One or more wideband signals observed from these structures and their surrounding areas can be used as values of the updated reference matrix, which will be used in the subsequent decomposition operation. Furthermore, the shape and intensity information of the structures in the region of the image, together with the biological criteria, can be used in determining the confidence for each structure. The detected structures can be scored to generate a confidence, and a reference spectrum from only the reliable structures is used in the subsequent resolving operation. Further embodiments disclosed herein perform operations including ranking detected structures, such as nuclei, based on one or more criteria. For example, a subset of detected nuclei scored with a confidence higher than a threshold may be accepted as a true positive. Observe these approved structures and automatically estimate the updated reference spectrum. For example, the values of the updated reference spectrum from the selected kernel can be replaced with the initial DAPI reference spectrum in the updated reference matrix. Thus, subsequent decomposition operations using the updated reference matrix will yield improved results.

  [0024] The reference spectrum estimation and subsequent decomposition may be repeated, and the reference spectrum may be iteratively updated until an optimized reference matrix is obtained. Subsequent decomposition can be performed on all nuclei in the image or region thereof, or on reliable nuclei, or a separate subset of nuclei, and any combination thereof. . The determination of the optimized reference spectrum may be dependent on the threshold number of iterations, or the convergence or stability of the reference matrix, or any combination thereof. The optimized reference matrix leads to a more reliable decomposition of hyperspectral data than in the case of existing methods. For example, a resolving operation using an optimized DAPI reference spectrum can generate sharper contours between the nucleus and surrounding tissue.

  [0025] Further, subsequent decomposition operations can be performed iteratively on the same or different regions of the image, or the entire image. For example, a sparse grid (ie, a grid with a long distance between any two vertices) can be provided on the image, structures within a certain distance from one or more grid vertices, an estimated reference spectrum Can be sampled for Custom regions can be defined based on observed structures or features in the image, and separate optimized reference matrices for each region allow images with various wideband signatures to be analyzed. Different operations can be performed in parallel for different regions, which allows for the efficient processing of multiple multiplexed fluorescent slides. Subsequent decomposition operations can be performed on all structures in the region, or on separate subsets of structures, based on the reliability of the structure and surrounding regions and other biological criteria.

  [0026] FIG. 1 shows a system 100 for adaptive spectral decomposition in accordance with an exemplary embodiment of the present subject disclosure. System 100 comprises a source 101 for generating an image, such as a fluorescence image. Other types of images, such as immunohistochemistry (IHC: immunohistochemical) bright field imaging with hematoxylin / eosin and other stains are also within the scope of the present disclosure. For example, source 101 can be a fluorescence microscope that produces a fluorescence image or a bright field microscope that produces an RGB image. Source 101 is in communication with memory 103, which contains a plurality of processing modules or logical operations, which are executed by processor 105 connected to interface 107. For example, a sample, such as a biological specimen, can be mounted on a slide or other substrate or device for imaging with a microscope connected to memory 103, and analysis of the image of the specimen is a memory according to the present disclosure This may be done by the processor 105 executing one or more of the plurality of modules stored at 103. The analysis may be for sample identification or investigation. For example, biological or pathological systems may be used for biological information such as proteins, protein fragments or other markers indicative of cancer or other illness, or for other purposes such as genomic DNA detection, messenger RNA The specimens can be examined for detection, protein detection, virus detection, gene detection, or the like.

  [0027] The preparation can be stained by applying one or more different stains, which may include one or more different quantum dots, fluorophores, or other stains. The number N of stains, such as fluorochrome molecules, applied to the specimen may vary. The fluorochrome molecules can include one or more nanocrystalline semiconducting fluorochrome molecules (ie, quantum dots), each producing a peak emission response within a different wavelength range. Quantum dots are well known and may be commercially available from Invitrogen, Evident Technologies, and others. For example, the sample can be processed using several different quantum dots that produce peak emission responses of 565, 585, 605, and 655 nm, respectively. One or more of the fluorochrome molecules applied to the specimen can be organic fluorochrome molecules 14 (eg, DAPI, Texas Red), which are well known in the art and are at least U.S. Pat. No. 8,290,236, commonly-owned and commonly-owned, which is hereby incorporated by reference in its entirety. Moreover, typical specimens are treated with an automated staining / assay platform that applies stains comprising quantum dots and / or organic fluorescent dye molecules to the specimens. There are a variety of commercially available products suitable for use as a staining / assay platform, an example being DiscoveryTM automated slide staining platform from assignee Ventana Medical Systems.

  After preliminary tissue processing and staining, the specimen is provided to the image acquisition module 111 to generate a digital image of the specimen observed at the source 101. The image acquisition module 111 generates a luminescence response from, for example, a scanner or a spectral camera used to image a slide containing a sample of a substance stained with a fluorescent stain and a fluorochrome molecule applied to a specimen It can be connected to a light source for illuminating the sample at the intended wavelength. In the case of quantum dots, the light source can be a broadband spectral light source. Alternatively, the light source can comprise a narrow band light source such as a laser. Also, the camera platform can include a microscope with a set of one or more objective lenses and digital imagers and spectral filters. Other techniques for capturing images of different wavelengths may be used. Camera platforms suitable for imaging stained biological specimens are known in the art and are commercially available from companies such as Zeiss, Canon, Applied Spectral Imaging, and others, and such platforms are the subject of the present application. Easily adapted for use in the subject system, methods, and apparatus. Images may be computer, via a cable connection between source 101 and computer 107, via a computer network, or using any other medium commonly used to transfer digital information between computers. The readable medium 103 can be supplied. Images may also be provided over a network to a network server or database for storage and later retrieval by computer 107. In addition to processor 105 and memory 103, computer 107 further includes input and output devices, such as a keyboard, a mouse, a stylus, and a display / touch screen. As described in the following discussion, processor 105 executes the modules stored in memory 103 to analyze images, morphologically process images or image data derived from such images, quantitative analysis, and users. Display of quantitative / graphical results on the operation computer 107.

  [0029] For example, as mentioned above, the slide holding the specimen is observed at the source 101, and the scanned image comprising a mixture of several fluorescence channels comprising one or more broadband signals is mentioned above. It is generated by the image acquisition module 111. Region selection module 112 allows automatic or manual delineation of images into one or more regions. This allows subsequent decomposition operations to be performed on the same or different areas of the image, so that multiple areas or multiple images of an image can be processed efficiently. For example, a grid such as a sparse grid can be provided on the image, and structures that are within a certain distance from one or more grid vertices are sampled for an estimated reference spectrum, as described further herein. can do. The grid size can be optimized for speed or for detailed processing and can be defined by the user of the system. Custom regions can be defined based on observed structures or features in the image, and separate optimized criteria for each region allow images with different broadband signatures to be analyzed. Custom regions can be automatically determined based on image analysis, tissue heterogeneity, etc., or can be selected by the user. Different operations can be performed in parallel in different regions, which allows for efficient processing of multiple multiplex slides, for example fluorescent slides. A region can be selected that contains predominantly or purely broadband signals in the absence of quantum dot signals to determine an initial or estimated reference spectrum as described herein.

  [0030] The spectral decomposition module 114 may be implemented to decompose images or selected regions of the image and separate component fluorescent channels using the non-negative least squares method presented herein. The image may comprise a mixture of end member spectra, including DAPI and AF, and one or more quantum dots, as described above, which are first resolved using the initial reference spectra. The initial reference spectrum can be taken from the control image or can be estimated from the image under observation. The initial reference spectrum can be retrieved from a database including one or more reference matrices, such as the reference spectrum database 113. The initial reference spectrum can be obtained from the reference matrix used in the resolving operation.

  [0031] Additionally, an initial wideband reference signal or initial reference signal can be determined by observing the ubiquitous distribution of the unique signature over the image or area of the image. For example, a ubiquitously distributed wideband signal can be identified as a DAPI signal based on a comparison of its signature with a known signature from a known wideband signal (ie, DAPI sampled from a control slide) . As mentioned above, one or more initial reference spectra can be obtained from the control image or determined from the image being observed. The initial reference spectrum can be estimated based on a comparison of the measured wideband signature and the known wideband signal stored in the reference spectrum database 113. One or more wideband signatures in the image can be recognized by its unique signature and the ubiquitous distribution across the image. A particular area of the image may be determined to include predominantly or only a broadband signal such as autofluorescence. The profile of these wideband signals as well as the ubiquitous distribution throughout the image support the assumption that these recognized signals can be used as an initial reference spectrum for the decomposition. Furthermore, once the component signal with the broad band signature is determined, the component signal can be compared to known broad band signatures specific to the stain and / or the specimen being analyzed. For example, a system for dissection or clinicopathology compares a scanned slide of a tissue sample with an image of a calibration slide containing a similar tissue sample having a known broad band signature, in the scanned image. Broadband signals can be identified.

  [0032] The spectral decomposition module 114 decomposes the image and / or component signals of the area, and the weight of each stain is given, for each image pixel, the spectral signature of each pixel as discussed above It is calculated using the initial reference spectrum determined in. The initial reference spectrum can be stored in an array or matrix comprising a plurality of known signals, including broadband and narrowband signals corresponding to fluorochrome molecules in the image. For example, each column of the reference matrix can represent a reference spectrum that corresponds to a particular stain or fluorophore. Applying a matrix to decompose the image allows linear spectral decomposition processing to be used to extract one or more signals. The spectral signature of a single pixel in the multi-channel image is obtained as a linear combination of the spectral signatures of all the different fluorochrome molecules, each signature being weighted with the corresponding weight of each fluorochrome molecule at that pixel. In multi-channel images, the individual weights for each stain's binding can not be accessed, but the spectral signature of each pixel can be retrieved. The set of extracted spectral or target signals can be reconstructed to generate an image suitable for analysis without containing noisy or unwanted spectra. In addition, wideband channels resolved from the image can be processed to estimate a reference spectrum that has been optimized as described herein.

  [0033] The structure detection module 115 may be implemented to perform operations including detecting and segmenting structures such as nuclei from broadband degradation channels, tissue structures, and their boundaries. Structure detection operations include nuclear detection and segmentation. For nuclear detection, a radial symmetric voting method can be applied to determine the location of the nuclei in the image or selected area. The magnitude of the gradient can be calculated from the decomposition results of the spectral decomposition process, and for each pixel around a specified size, some votes based on the sum of that size in the area around that pixel It can be assigned. The formula for calculating the magnitude of the slope is

Given as where I is the image,

Is the gradient in the x direction,

Is the gradient in the y direction. A mean shift clustering operation can be performed to find a local center in the voting image, which represents the actual position of the nucleus. The nuclear segmentation operation segments the entire nucleus by performing morphological operations and local thresholding using the currently known centers of the nuclei. Model-based segmentation and other operations can also be performed depending on processing power and time requirements. For example, the nuclear segmentation operation can further include learning a shape model of the nucleus from the training data set and segmenting the nuclei in the test image using this as prior knowledge.

  [0034] A ranking module 117 is performed to perform operations including determining confidence for each detected structure based on multiple factors including shape and strength information and biological criteria. These criteria can include the requirement that selected regions of the structure or image not be co-located with other biomarkers except quantum dots or broadband signals. In other words, a pure broadband signal is preferred, which signal is as discussed above, ie coincident with a similar known reference spectrum for DAPI, and / or localized across the entire area of the image or image. Identified based on the variance. Structures such as nuclei that overlap or are in the same position as other stained structures such as tumor or tissue markers can be rejected or ranked low by examining pixel intensity. Some nuclei may be rejected. For example, nuclei which are extremely elongated or narrow as determined by the ratio of the major and minor axes, which are smaller or larger than a given threshold, are rejected. One exemplary threshold value for the ratio may be two, ie, any ellipse with a major axis twice as long as the minor axis is rejected. An ellipse fitting operation can be performed to fit an ellipse over each segmented kernel, using the dice and the die coefficient of the segmented kernel as described further herein. Score with confidence. The detected structures can be ranked in order of their score or confidence determined by their dice factor and other criteria. For example, a subset of detected nuclei scored with a confidence higher than a threshold may be accepted as a true positive.

  [0035] As described herein, the reference matrix may be updated with the reference spectrum estimated from the highest ranked structure by the reference spectrum estimation module 118, and the updated reference matrix may be decomposed or Used in subsequent disassembly operations. Reference spectrum estimation module 118 observes broad band signatures, such as from regions of the image corresponding to the selected or highest quality structures, and estimates the reference spectra for these regions. One or more wideband signals observed from these structures and their surrounding areas can be used as the values of the updated reference matrix, which will be used in subsequent decomposition operations performed by the spectral decomposition module 114. be able to. For example, values of a reference spectrum estimated from a high rank nucleus (i.e., nuclei identified as having high confidence values) can be replaced with the initial DAPI and / or AF reference spectra in the reference matrix. Thus, subsequent decomposition operations using the generated reference matrix yield improved results. Reference spectrum estimation can be performed for a plurality of structures within the image or region of the image, and using the median pixel value of the reliable structure region within the multispectral image to determine the updated reference spectrum Can. The updated reference spectrum can be added as a value to the reference matrix stored in database 113 and used in subsequent decomposition operations.

  [0036] Furthermore, the reference spectrum can be iteratively generated or updated until an optimized reference matrix is obtained. The reference matrix optimization module 119 determines whether the updated reference matrix has been optimized. For example, the reference matrix can be updated iteratively through the iteration cycle of the decomposition 114 and the reference spectral estimate 118, with the result that each reference results in further optimization of the reference matrix for the image or region of the image. The optimization can be completed upon determining the convergence or stability of the reference matrix. Alternatively or additionally, execution of a threshold number of iterations can be monitored to determine optimization. The optimized reference matrix results in improved decomposition results of hyperspectral data better than with other methods. For example, subsequent decomposition operations using the optimized DAPI reference spectra stored in database 113 can generate sharper contours between the nucleus and surrounding tissue. Furthermore, the optimization can be repeated for different regions of the image, each region having a separate optimized reference spectrum.

  As mentioned above, the modules include the logic executed by the processor 105. As used herein and throughout the present disclosure, "logic" refers to any information having the form of instruction signals and / or data available to affect the operation of the processor. Software is an example of such logic. Examples of processors are computer processors (processing units), microprocessors, digital signal processors, controllers, microcontrollers and the like. Logic may be formed from signals stored on a computer readable medium such as memory 103, such as random access memory (RAM), read only memory (ROM), erasable / electrically erasable programmable read It includes dedicated memory (EPROM / EEPROM), flash memory and the like. The logic may further comprise digital and / or analog hardware circuits, eg hardware circuits comprising logical AND, OR, XOR, NAND, NOR and other logical operations. Logic can be formed from a combination of software and hardware. On a network, logic can be programmed on a server or collection of servers. A particular logical unit is not limited to a single logical location on the network.

  [0038] FIG. 2 illustrates a method for adaptive spectral decomposition in accordance with an exemplary embodiment of the present subject disclosure. The method of FIG. 2 can be implemented by a computer executing modules similar to those shown in FIG. The method can capture image content at a source, such as a fluorescence microscope associated with or including a scanner or spectral camera, or hyperspectral or fluorescence where image energy is captured at multiple frequencies Starting with the image or image data of the specimen received from any source that allows imaging (S220). The specimen can be stained by applying one or more different stains including, for example, quantum dots or fluorochrome molecules illuminated by a light source. Following staining, an image is captured by a detection device such as a spectral camera as described above. The image is provided to a computer, which executes the logical instructions stored in the memory to perform the operations described in the exemplary method.

  [0039] For example, the image or the region thereof can be decomposed to extract a wide band signal such as DAPI from the image (S221). The decomposition can use the non-negative least squares method presented herein to separate the component fluorescent channels of the image or region thereof. Images can be drawn automatically or manually into one or more regions. This makes it possible to process multiple images efficiently, since subsequent decomposition operations can be performed on the same or different regions of the image. The initial reference spectrum can be taken from the control image, or can be estimated from the image being observed as described herein. The initial reference spectrum may be the value of a reference matrix comprising a plurality of reference signals applied to the mixture of observed signals, which results in a plurality of decomposed component signals of the image. For example, a matrix in which a reference spectrum and a plurality of narrowband signals corresponding to a target signal are input can be applied to the mixture of signals, resulting in a set of component signals.

[0040] To emphasize one exemplary non-negative least squares decomposition operation, let Am _{x n} be an end member reference spectral matrix, and each column of A corresponds to an end member spectrum that is an m-dimensional vector. A = [a _d , a _a , a _{1. . . , K 1} ] are the spectra for DAPI, AF, and quantum dots, respectively. n is the number of biomarkers / end members. Y _{m × N} is a multispectral image comprising N pixels, each pixel being an m-dimensional vector imaged from m spectral narrowband filters. The decomposition equation shown below is formulated to minimize the overall least squares error:

Here, x _i is an n-dimensional vector of the abundance corresponding to the ith pixel of the decomposed image, and y _i is the ith pixel of Y. By solving this equation, n end members X _{n × N} = [x ₁ , x ₂ ,. . . , X _N ] is obtained. Further details regarding non-negative least squares solution processing can be found in Lawson, C. et al. L. And R. J. It can be found in Hanson, Solving Least Squares Problems, Prentice-Hall, 1974, chapter 23, page 161.

  [0041] Within the resulting set of component signals, there is at least one wideband signal, which is analyzed for the presence of structure (S223). This step includes detecting and segmenting structures such as nuclei from at least one broadband channel data of the image, as well as identifying tissue structures and their boundaries. Structure detection operations include nuclear detection and segmentation. For nuclear detection, a radial symmetric voting method can be used to determine the position of the nucleus in the image or selected area. The size of the image can be calculated from the decomposition results of the spectral decomposition process, and each pixel around the specified size is assigned several votes based on the sum of that size in the area around that pixel be able to. A mean shift clustering operation can be performed to find the local center in the voting image, which represents the actual position of the nucleus. Nuclear segmentation uses the currently known centers of the nuclei to segment the entire nucleus by performing morphological operations and local thresholding. Model-based segmentation and other operations can also be performed depending on available processing power and time.

  [0042] Noise in the image and potentially inaccurate reference spectra can lead to false positive results among the detected structures. Thus, a ranking operation is performed to determine the confidence level for each detected structure based on multiple factors including shape and intensity information and biological criteria. These criteria ensure that the selected regions of the structure or image are not co-localized with the quantum dots, or other than the broadband signal, by confirming the presence of the signal at the same position for both DAPI and quantum dot images. Include the requirement of not co-localizing with the biomarker. Extremely elongated structures are rejected as determined by the ratio of the major and minor axes that are smaller or larger than a given threshold. An ellipse fitting operation can be performed to fit an ellipse over each segmented kernel, and the dice are used to score the nuclei with confidence. The detected nuclei can be ranked in order of score or confidence determined by their dice factor and other criteria. For example, a subset of detected nuclei scored with a confidence higher than a threshold may be accepted as a true positive.

  [0043] The reference spectrum can be estimated from the highest ranked structure (S225). This step involves observing the wideband signatures, such as DAPI, from the highest ranked structures in the predefined regions of the image, and estimating the updated reference spectra for these structures and their surrounding regions. The derived signal may be used to update or generate a reference matrix (S227), which may be used in subsequent decomposition operations. For example, the value of the updated reference spectrum estimated from the high rank kernel can be replaced with the initial DAPI reference spectrum in the updated reference matrix. Thus, the decomposition operation with the generated reference spectral matrix / data, or the subsequent decomposition operation with the updated reference matrix, leads to improved results. Reference spectrum estimation (S225) may be performed on the image or multiple structures within the region of the image, and the median broadband signal may be used to determine the updated reference spectrum (S227). The updated reference spectrum can be added as a value to the reference matrix stored in the reference database and used in subsequent decomposition operations.

[0044] Further, the reference spectrum can be updated iteratively until an optimized reference matrix is obtained (S229). Further decomposition (S221) processing can be triggered by a determination to see if the maximum number of iterations has been exceeded or the updated reference matrix has been optimized, including structure detection (S223), reference spectral estimation. (S225) and update (S227) follow, and as a result of each iteration, the reference matrix for the image or region of the image is improved. The optimization (S229) may be completed upon determining the convergence or stability of the reference matrix. By monitoring the relative difference of each updated reference spectrum between two consecutive iterations, convergence can be determined. Alternatively or additionally, the execution of a threshold number of iterations can be monitored to determine optimization, one exemplary threshold number being t = 100. One exemplary loop for optimal reference spectrum optimization (S229) may include the following operations:
1: Initial reference spectral matrix A = [ _ad , _aa , a _{1. . . , K} ] and the multispectral image Y are input 2: for t <maximum iteration number or || A _{t + 1} −A _t || ² ≦ ∈ do
3: Solve for x:

4: to estimate the broadband reference spectrum _{a d} and _{a a} for DAPI and AF (step _S225 in FIG. ^{_{2), A t + 1 =}} [a d new, a a new, a 1. _{. . , K} ] to update A _t 5: end for
[0045] The optimized reference matrix results in a more reliable resolution of hyperspectral data that is improved compared to other methods. Furthermore, the optimization can be repeated for different regions of the image, each region having a separate optimized reference spectrum. If all required reference spectra of the reference spectral matrix have been optimized (S 229), the method ends and the image can be subjected to further processing.

  [0046] FIGS. 3A and 3B illustrate an image 330 segmented into regions for adaptive spectral decomposition, according to an exemplary embodiment of the present disclosure. The image 330 can be rendered on a computer display and can be generated by an application such as one of the modules shown in FIG. Image 330 displays a specimen scanned from a slide and depicts a plurality of structures 331. The specimens may, for example, take the form of tissue sections obtained from human or animal subjects, such as formalin fixed paraffin embedded tissue samples. The preparation may be biological cell tissue, frozen cells, tumor cells, blood, pharyngeal culture, or others, the type or nature of the preparation being less important. The structure may be a tissue cell, a nucleus, a cancer cell, or any other structure.

  [0047] As described herein, the image is delineated into one or more regions, which allows parallel decomposition and optimization operations to be performed in different regions. This can be achieved by overlaying the sparse grid on the image 330, as represented by the dashed line 333 shown in FIG. 3A. Structures within a certain distance from one or more grid vertices, such as nuclei 331 in region 335, can be sampled for the estimated reference spectrum. This allows different parts of the image to be processed independently to estimate the median reference spectrum for the entire image. Alternatively or additionally, the regions delineated by line 333 can be processed independently to determine separate optimized reference spectra for each region.

  [0048] Additionally, as shown in FIG. 3B, custom regions can be defined based on the observed structures or features in the image. This custom segmentation makes it possible to generate separate optimized reference spectra for each region, in particular for images with fluctuating DAPI channels. Custom segmentation can be determined automatically based on the identification of structures in the image, ie, using pixel intensity variations, tissue non-uniformity, etc. For example, the nuclei within the tumor area may be larger, thereby separating the tumor area from other areas of the image. Local histograms can be used to group and delineate similar regions, and subsequent detection and optimization processes are performed independently for each region. The custom region may be user selectable via the user interface. For example, DAPI or broadband areas can be identified and selected by the user. In addition, region selection can be done anytime during processing, including after identification of trusted structures.

  [0049] FIG. 4 illustrates a method for detecting structure in a wideband signal of an image according to an exemplary embodiment of the present disclosure. The method of FIG. 4 can be implemented by a computer executing modules similar to those shown in FIG. The method starts with an initial decomposition operation (S440) using an initial reference spectrum, wherein at least one decomposed broadband signal is obtained. Regions of the image can be selected (S441) for nuclear detection (S445) and segmentation (S447). For example, the image can be automatically or manually delineated into one or more areas as described herein. Structures that are within a certain distance from one or more grid vertices can be sampled for the estimated reference spectrum, as shown in FIG. 3A. Once the image area is selected, morphological operations can be performed (S 443) to remove background noise and to determine the kernel foreground. The center of each nucleus can be determined using a method such as radial symmetry voting (S445). For example, given a decomposed broadband channel, the size of the image is calculated from the channel and one or more votes of each pixel are accumulated by summing their sizes in the selected area Can. Mean-shift clustering can be used to find local centers in the region, where the local centers represent the actual nuclear location.

  [0050] The detected nuclei can be further segmented from surrounding tissue (S447). Segmentation is performed using morphological centers (for example, dilation to enlarge the area and erosion to remove isolated noise and shrink the area) using the nucleus center determined in S445. The surrounding tissue is acquired by performing in the area adjacent to the nucleus. Local thresholding can be performed to segment the entire nucleus. Because the intensity of the kernel pixel is higher than that of its neighbors, the intensity cutoff / threshold can be used to separate the kernel and nearby background areas. Model-based segmentation and other operations, such as pixel-by-pixel classification using a K-mean method may also be performed. Once the nuclei and surrounding tissue have been detected and segmented, an ellipse fitting operation (S449) can be performed on each nucleus to allow subsequent confidence determination and ranking operations. As most nuclei are circular and / or elliptical, a robust ellipse fitting algorithm based on random sample consensus (RANSAC) can be implemented to fit an ellipse over each segmented nucleus. By fitting an elliptical mask to each nucleus, subsequent confidence determination operations can be performed. One exemplary ellipse fitting method is:

As a landmark along the boundary of the nucleus. Sum of squared algebraic distances

To minimize the parameters of the elliptic model

Solve about, where

And

Represents the parameter of the ellipse.
[0051] FIG. 5 illustrates a method for determining the confidence of a detected structure according to an exemplary embodiment of the present subject disclosure. Confidence can be based on multiple factors including shape and strength information as well as biological criteria. The method starts with the result of the above-mentioned nuclear component detection processing S550, and the detected nuclei are segmented and fitted with an elliptical mask. Each nucleus is observed independently, and an ellipse is selected before performing the reliability determination (S551). For example, the area of the ellipse is calculated (S553), and if the area is larger or smaller than the threshold, the ellipse can be rejected (S559). This action further includes determining the elongation of the ellipse, ie the ratio of the major axis to the minor axis of the ellipse, and the ellipse will be rejected if any ratio is above or below the threshold ( S559). For example, the maximum threshold may be two. If the area and elongation of the ellipse meet the threshold standard, then the dice factor shown below:

An ellipse fit is scored using (S 555), where V is an area operator, p is a binary image of the segmented kernel, and q is an ellipse mask.

  [0052] Nucleus can be scored and ranked based on dice factor (S555), and scores below threshold can be rejected (S559). In one exemplary embodiment, the threshold dice ratio is 0.9. For ellipses that meet this threshold (ie, have a score of 0.9 or greater), additional criteria may be applied (S557) before being approved, eg, to meet biological constraints. These can include requiring that the detected structure not be at the same position as, or overlap with other biomarkers other than quantum dots or broadband signals. If these conditions are not met, the ellipse may still be rejected (S559). If these conditions are met, the method determines if there are more ellipses in the area to be scored (S558). If there is a further ellipse, the method starts again with the selection of the next ellipse (S551). If no more ellipses exist, the method can continue to perform further operations, such as ranking and estimating the reference spectrum from the approved structure.

  [0053] FIG. 6 illustrates the ranking of nuclei using an elliptical fit, according to an exemplary embodiment of the present subject disclosure. Image 630 shows a specimen scanned from a slide containing a plurality of stained structures 631. For example, the specimen may comprise tissue sections, such as formalin fixed paraffin embedded tissue specimens, and may have been obtained from a human or animal subject. The preparation may be biological cell tissue, frozen cells, tumor cells, blood, pharyngeal culture, or others, the type or nature of the preparation being less important. The structure may be a tissue cell, a nucleus, a cancer cell, or any other structure. An ellipse 637 is fitted to the structure 631 and each ellipse can be scored and ranked based on multiple thresholds applied to the ellipse as described herein. For example, an ellipse with a die coefficient less than 0.9 can be rejected as marked as "R" in FIG. Ellipses having a score higher than 0.9 and meeting additional biological criteria can be accepted and marked as "A". In addition, a threshold can be applied to the elongation of the ellipse, and on the assumption that most nuclei are close to spherical, extremely elongated ellipses are rejected. Intensity constraints can also be applied to detected nuclei prior to approval or rejection. For example, a nucleus may not be approved if the intensity of the nucleus pixel is less than 0.1 as determined by the mean intensity-2 × intensity standard deviation.

  [0054] Furthermore, autofluorescent regions of tissue are generally found near the nucleus. With the highly reliable nucleus marked as approved, a small neighborhood within a few pixels of the periphery of the nucleus, as shown by the ring 638 around the fitted ellipse 637, Designated as an area. Since the nuclear ranking procedure guarantees segmentation of the nuclear and tissue regions, a broad spectrum including DAPI and AF can be estimated from these regions, and the reference spectral matrix A is generated or updated accordingly. The subsequent decomposition and estimation steps described above can be applied recursively until the broadband reference spectrum stabilizes or exceeds the maximum number of iterations. The median reference spectra for the image or selected regions of the image can be stored in the optimized reference spectral matrix.

  [0055] FIG. 7 illustrates a graph identifying median reference spectra estimated from a plurality of high rank structures, according to an exemplary embodiment of the present disclosure. Each nucleus is estimated to have a reference spectrum identified by dashed line 760. For a particular region of the image, the median value of the reference spectrum for each nucleus in that region is calculated and illustrated by solid line 761, which is the median value of all dashed lines 760. This is used as the generated reference spectrum for that region in the updated reference matrix A. Alternatively or additionally, FIG. 7 can represent a plurality of reference spectra 760 determined for the corresponding plurality of iterations of the optimization process, and the resulting median reference spectrum 761 represents the optimized reference spectra. . The image is properly resolved using the optimized reference spectrum to provide improved results.

  [0056] FIG. 8 illustrates a comparison of nuclear detection results using an initial reference spectrum and an updated reference spectrum, according to an exemplary embodiment of the present subject disclosure. The resulting image 871 shown in the left column shows the detected nuclei in a broadband channel that has been resolved using a standard reference spectrum derived from a control image. The results 872 in the right column show nuclei detected after the adaptive decomposition process described in the disclosed embodiment. It is clear that in each region there is an improved nuclear detection result with an optimized reference spectrum compared to a standard reference spectrum.

  [0057] Observed spectral image or hyperspectral image data set and the existing method according to the disclosed image analysis operation, eg, improved estimation of the broadband reference spectrum obtained using the improved identification of cells Image-specific estimates of broadband signatures such as AF and DAPI are obtained while minimizing the differences in the estimation matrix. Other broad band signatures, such as red blood cell signatures, can also be optimized. The disclosed systems and methods are highly suitable for blood vessel detection and macrophage segmentation because the method provides more accurate spectral resolution results and sharper, sharper DAPI channels. Both vascular detection and macrophage segmentation require an accurate estimation of the nuclei (stained by DAPI) as the first step. The method of the present invention is also applicable to in situ hybridization (ISH) images. ISH is a useful technique for spatially localizing specific probes / targets in tissues and cells, providing information on gene expression and gene loci. The probe may be in the form of a fluorescent marker (in the case of FISH etc.) or a chromogenic marker (in the case of CISH etc.). In both cases, the adaptive decomposition method of the present invention can be applied to FISH or CISH images to obtain, for example, improved estimates of nuclear marker signatures. Due to staining variations, the signature for the nucleus may change, and the best estimate of this nuclear channel may differ significantly from image to image. However, the adaptive decomposition method of the present invention may be for DAPI estimation in fluorescence images, but may also be used to identify top ranked shape groups of cells in FISH and CISH images. . Furthermore, statistical averaging over cells of these shapes (e.g. calculation of the median of the reference spectra of top ranked shapes of cells according to the present invention) results in improved estimates of the reference spectrum or reference color of the nucleus. Given. Thus, the adaptive decomposition method of the present invention uses physically meaningful structures to obtain more reliable estimates of the reference vector, but both bright field and fluorescence images, eg chromogenic It is applicable to bright field or dark field IHC images, chromogenic ISH bright field or dark field images, fluorescent ISH images, and / or quantum dot images. In addition to medical applications such as autopsy or clinicopathology, prostate / lung cancer diagnosis, the same method can be performed to analyze other types of samples, for example remote sensing of geology or astronomy data. By removing known or apparent noise sources, the image can be further purified, for example, as compared to a set of known or ideal signals from similar materials. Other refinements include adjusting the minimum or maximum intensity to highlight a particular range and removing signals outside that range, and adjusting the contrast to see more dynamic range And other imaging operations. The operations described herein are ported to a hardware graphics processing unit (GPU) for analysis of large or multiple slides / images, or to analyze one or more image cubes. This allows for multi-threaded parallel implementations.

  [0058] The foregoing disclosure of the exemplary embodiments of the present subject disclosure has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure of the present subject matter to the precise form disclosed. Many variations and modifications of the embodiments described herein will be apparent to one of ordinary skill in the art in light of the above disclosure. The scope of the present disclosure is to be defined only by the claims appended hereto, and their equivalents. Further, in describing representative embodiments of the present subject disclosure, the specification may have given the disclosed method and / or process of the present subject matter as a particular sequence of steps. However, the method or process should not be limited to the particular order of steps recited, as long as the method or process is not dependent upon the particular order of steps recited herein. As one skilled in the art will appreciate, other sequences of steps are possible. Therefore, the particular order of the steps set forth in the specification should not be considered as limitations on the claims. In addition, the claims directed to the methods and / or processes disclosed in the present subject matter should not be limited to the implementation of the steps in the order described, as the order may be changed and still be Those skilled in the art can easily understand that they are within the spirit and scope of the subject disclosure.

Claims (20)

Decomposing a first region of an image comprising a mixture of signals using a reference matrix comprising an initial reference vector to generate a decomposed first region of the image;
Estimating an updated reference spectrum for the resolved first region and generating an updated reference vector for the resolved first region;
Updating the reference matrix with the updated reference vector, the computer executable instructions being executed by a processor to perform an operation, wherein the updated reference vector is used in a subsequent resolving operation. Non-transitory computer readable medium for storage.   The non-transitory computer readable medium of claim 1, wherein the initial reference vector is derived from a control image.   The non-transitory computer readable medium according to claim 1, wherein the updated reference vector is estimated from a plurality of structures detected within the resolved first region of the image.   4. The non-transitory computer readable medium of claim 3, wherein the plurality of structures are detected by a combination of radial symmetry voting and segmentation between a plurality of nuclei and a plurality of tissue cells.   The operation is based on biological criteria including at least one of: a region within pure 4 ', 6-diamidino-2-phenylindole (DAPI) region, a hematoxylin region, non-overlapping nuclei, a size threshold and an elliptical fit 5. The non-transitory computer readable medium of claim 3 or 4, further comprising the step of: ranking the plurality of structures.   A non-transitory computer readable medium according to any of claims 3 to 5, wherein the operation further comprises determining a degree of confidence for each of the plurality of structures.   The non-transitory computer readable medium of claim 6, wherein the operation further comprises: estimating the updated reference vector from a subset of the plurality of structures meeting a confidence threshold. The operation is
Determining a degree of confidence for each of the plurality of structures;
Estimating the updated reference vector from a subset of the plurality of structures meeting a confidence threshold
Further include
6. The non-transitory computer readable medium of claim 5 , wherein the confidence threshold is based in part on a die ratio for the elliptical fit.   The non-transitory computer readable medium of claim 7, wherein the operation further comprises: estimating the updated reference vector from an area adjacent to one or more structures of the subset of the plurality of structures.   10. A non-transitory computer readable medium according to any of the preceding claims, wherein the operation further comprises iteratively decomposing the image and updating the reference matrix until an optimal reference spectrum is obtained.   11. The non-transitory computer readable medium of claim 10, wherein the optimal reference vector is obtained if the reference matrix is stable and / or if a threshold number of iterations has been reached.   12. The any one of claims 1 to 11, wherein the resolved channel comprises at least one of 4 ', 6-diamidino-2-phenylindole (DAPI) signal, autofluorescence signal, and hematoxylin (HTX) signal. Non-transitory computer readable medium according to claim 1. A system for decomposing an image,
A processor,
A memory connected to the processor, the memory comprising:
Estimating a reference vector for a stain from a high rank subset of structures detected from the resolved signal corresponding to the stain;
Decomposing the image using a reference spectrum, the estimating and decomposing steps being repeated until an optimal reference spectrum is obtained, when the processor performs the operation on the processor A system that is for storing computer-executable instructions that cause it to occur.   The system of claim 13, wherein the plurality of structures comprises a plurality of nuclei.   15. The system of claim 14, wherein the high rank subset of the plurality of structures comprises a subset of the plurality of nuclei satisfying a confidence threshold.   16. The method of claim 15, wherein the operation further comprises delineating the image into a plurality of regions, wherein the high rank subset of the plurality of nuclei is located within a first region of the plurality of regions. system.   17. The system of claim 16, wherein the operation further comprises: resolving a second region of the plurality of regions using the reference vector. A method for disassembling,
Decomposing an image containing a mixture of signals to generate a decomposed signal;
Detecting a plurality of structures in the decomposed signal;
Ranking the plurality of structures in order of reliability;
Estimating a reference vector for a stain, as indicated in the image, from a subset of structures in the plurality of structures having a reliability higher than a threshold reliability;
Storing the reference vector in a reference matrix associated with the image;
And subsequently resolving the image using the reference matrix.   19. The method of claim 18, further comprising repeating the steps of detecting, ranking, estimating, and subsequently resolving until an optimal reference vector is obtained.   20. The method of claim 19, further comprising determining convergence of the reference matrix to obtain the optimal reference vector.

Images