JP2013113818A - Image processing device, microscope system, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device and the like capable of giving order depending on order in which an original biological tissue sample is sliced, to a series of sample images obtained by imaging slice samples of the biological tissue sample.SOLUTION: The image processing device comprises: a component image creation part 452 which extracts at least one kind of tissue component on the basis of feature quantity of color of each pixel in an image, from multiple images corresponding to multiple slice samples obtained by slicing a biological tissue sample and performing coloring, the component image creation part 452 creating component images each of which is an image for each kind; and a continuity evaluation part 453 which has continuity evaluation algorithms 453a, 453b, ..., for evaluating continuity of the component image of the same kind among the multiple images, the continuity evaluation algorithms 453a, 453b, ... determines order of the multiple images, on the basis of the evaluation result of the continuity.

Description

  The present invention relates to an image processing apparatus, a microscope system, an image processing method, and an image processing program for processing a specimen image of a biological tissue acquired by a microscope.

  Conventionally, in biological tissue specimens, especially pathological specimens, specimens obtained by organ excision, needle biopsy, etc. are sliced to a thickness of about several microns and then magnified with a microscope to obtain various findings. ing. In recent years, an observation image of a slice specimen obtained by slicing a specimen is converted into electronic image data, and a series of slice specimen images are three-dimensionally displayed and compared in an image processing apparatus. In many cases, a plurality of images are displayed side by side on one screen for diagnosis.

  For example, Patent Document 1 discloses that position data regarding the contours of a plurality of continuous slice images is taken and a stereoscopic image is reconstructed based on position data obtained by converting the position data into world coordinates.

  In Patent Document 2, a slice cross-sectional image is binarized, a skeleton image is created by distance conversion, and a slice cross-sectional image is aligned with feature points extracted based on the skeleton image to construct a three-dimensional image. It is disclosed.

  Patent Document 3 discloses that a background is removed from each slice image as a preprocessing when a stereoscopic image of continuous slice images is synthesized.

  Since a slice specimen obtained by slicing a biological tissue specimen is almost colorless and transparent, it is usually stained prior to observation. Many methods are known as staining techniques, but for pathological specimens, hematoxylin-eosin staining (hereinafter referred to as HE staining) using two kinds of pigments of blue-violet hematoxylin and red eosin is performed. Is common. In HE-stained specimens, cell nuclei and bone tissue are stained blue-purple, and cytoplasm, connective tissue, red blood cells, etc. are stained red, so that the observer can easily see these tissue components. become able to. Thereby, the observer can grasp the size and positional relationship of tissue components such as cell nuclei in the slice specimen, and can determine the state of the slice specimen morphologically.

  As a technique related to staining of a biological tissue specimen, Patent Document 4 discloses a spectrum (spectral transmittance) at each position of a specimen image based on the pixel value of each pixel of an image obtained by performing multiband imaging of a specimen subjected to HE staining. ), The amount of dye at the position is estimated based on this spectrum, and the amount of dye at each position is corrected based on the dye amount distribution.

JP-A-3-75508 JP-A 64-1081 JP 2000-293692 A JP 2009-14355 A

  By the way, a plurality of section specimens obtained from one biological tissue specimen are usually managed by the number of a label attached to a slide glass on which the section specimen is placed. In recent years, a barcode label indicating the order of the section specimens may be attached to the slide glass.

  However, the operation of slicing a biological tissue specimen to create a slice specimen is often carried out manually. For this reason, if the label number, barcode, etc. are not properly managed, the continuity of each section becomes unclear, and the specimen images cannot be acquired in the order in which the original biological tissue specimen is sliced. As a result, there is a problem that it is difficult to obtain a 3D image that accurately represents the original biological tissue specimen, and that it is difficult to arrange specimen images in an appropriate order for comparative diagnosis.

  The present invention has been made in view of the above, and for a series of specimen images obtained by imaging a slice specimen of a biological tissue specimen, an accurate arrangement order according to the order in which the original biological tissue specimen is sliced An object of the present invention is to provide an image processing apparatus, a microscope system, an image processing method, and an image processing program.

  In order to solve the above-mentioned problems and achieve the object, an image processing apparatus according to the present invention includes a plurality of images corresponding to a plurality of slice specimens obtained by slicing and staining a biological tissue specimen. Extracting at least one type of tissue constituent element based on the color feature amount of each pixel, and generating an element image that is an image of each type, and the same type of the element between the plurality of images It has a continuity evaluation part which evaluates the continuity of an image, and is provided with the arrangement order judgment part which judges the arrangement order of a plurality of above-mentioned images based on the evaluation result of the continuity.

  The image processing apparatus further includes an image selection unit that selects a plurality of images estimated to be continuous from the plurality of images as a continuous section candidate image group, and the arrangement order determination unit includes the continuous section determination unit. The arrangement order between the candidate image groups is determined.

  In the image processing device, the image selection unit is configured to extract a contour from the plurality of images, and between an arbitrary image of the plurality of images and another image based on the contour. A similarity determination unit that performs similarity determination, and an image extraction unit that extracts, as the continuous segment candidate image group, images determined to be similar to each other by the similarity determination unit.

  In the image processing apparatus, the image selection unit includes a background extraction unit that extracts a background from each image of the plurality of images, an arbitrary image of the plurality of images, and another image based on the background. A similarity determination unit that performs similarity determination between the images, and an image extraction unit that extracts images determined to be similar to each other by the similarity determination unit as the continuous segment candidate image group.

  In the image processing apparatus, the element image creation unit creates a color feature quantity acquisition unit that obtains a color feature quantity of a pixel in each image, and creates a distribution of the color feature quantity, and the distribution is defined as the at least one A class classification unit that classifies into at least one class corresponding to a type of organization component, and a pixel corresponding to a color feature amount belonging to each class, and creates an image for each class as the element image And an image creating unit.

  In the image processing apparatus, the color feature amount is a pigment amount of the pigment in a sample portion corresponding to each pixel position in the image, and the color feature amount acquisition unit is a pixel of each pixel in each image. A spectrum estimation unit that estimates a spectrum of a color component at each pixel position based on a value; and a pigment amount estimation unit that estimates the pigment amount in the sample part based on a spectrum of the color component, and the class The classification unit classifies the pigment amount distribution by clustering the pigment amount distribution in a color feature amount space including the pigment amount as a component.

  In the image processing apparatus, the color feature amount acquisition unit further includes a pigment amount correction unit that homogenizes a distribution of the pigment amount among the plurality of images, and the class classification unit is corrected by the pigment amount correction unit. And classifying the distribution of the dye amount.

  In the image processing apparatus, the continuity evaluation unit evaluates the continuity between the element images by applying a continuity evaluation algorithm defined for each tissue component.

  In the image processing apparatus, the continuity evaluation unit performs the continuity evaluation based on at least one of the characteristics of the tissue constituent element, the site of the biological tissue specimen, and the predetermined dye. It is characterized by that.

  In the image processing apparatus, the at least one tissue constituent element includes at least one of a cell membrane, a cytoplasm, a cell nucleus, a blood vessel, blood, and a cavity, and the continuity evaluation unit includes the cell membrane and the cell nucleus. And the continuity is evaluated based on an element image corresponding to any one of the blood and the cavity.

  The image processing apparatus further includes a three-dimensional image configuration unit configured to three-dimensionally configure the plurality of images based on a determination result by the arrangement order determination unit.

  The image processing apparatus further includes an alignment unit that performs alignment between the plurality of images based on the position of the tissue component in the element image.

  A microscope system according to the present invention includes the image processing apparatus, a microscope apparatus that generates an observation image of a plurality of section samples obtained by staining a section obtained by slicing a body tissue specimen with a predetermined dye, and the observation And an image acquisition unit that acquires a digital image corresponding to the image.

  According to the image processing method of the present invention, for a plurality of images corresponding to a plurality of slice specimens obtained by slicing and staining a biological tissue specimen, at least one kind of tissue based on the color feature amount of each pixel in the image An element image creation step of extracting component elements and creating an element image that is an image of each type, and evaluating the continuity of the same type of the element image between the plurality of images, and the continuity evaluation result And an arrangement order determination step for determining the arrangement order of the plurality of images based on the above.

  The image processing program according to the present invention provides at least one type of tissue based on the color feature amount of each pixel in the image, for a plurality of images corresponding to a plurality of slice samples obtained by slicing and staining a biological tissue sample. An element image creation step of extracting component elements and creating an element image that is an image of each type, and evaluating the continuity of the same type of the element image between the plurality of images, and the continuity evaluation result And causing the computer to execute an arrangement order determination step for determining the arrangement order of the plurality of images.

  According to the present invention, since the arrangement order between a plurality of images is determined based on the continuity between element images corresponding to the tissue composition image extracted based on the color feature amount in the image, It is possible to give an accurate arrangement order according to the order in which the original biological tissue specimen is sliced.

FIG. 1 is a block diagram showing a configuration of a microscope system according to Embodiment 1 of the present invention. FIG. 2 is a schematic diagram illustrating a configuration example of the microscope apparatus illustrated in FIG. 1. FIG. 3 is a block diagram illustrating a configuration example of the image selection unit illustrated in FIG. 1. FIG. 4 is a flowchart showing the operation of the microscope system shown in FIG. FIG. 5 is a diagram illustrating an example of a sample image that is a processing target of the image processing apparatus illustrated in FIG. 1. FIG. 6 is a flowchart showing the operation of the element image creation unit shown in FIG. FIG. 7 is a diagram showing a color feature amount space in which the pigment amounts of the pixels in the sample image shown in FIG. 5 are plotted. FIG. 8 is a diagram for explaining a dye amount correction method. FIG. 9 is a diagram showing an element image of the cell membrane extracted from the specimen image shown in FIG. FIG. 10 is a diagram showing an element image of a cell nucleus extracted from the specimen image shown in FIG. FIG. 11 is a diagram showing an elemental image of red blood cells extracted from the specimen image shown in FIG. FIG. 12 is a diagram showing an element image of a cavity extracted from the specimen image shown in FIG. FIG. 13 is a diagram showing an element image of a wrinkle of a section sample extracted from the sample image shown in FIG. FIG. 14 is a diagram showing an element image of dust extracted from the sample image shown in FIG. FIG. 15 is a flowchart showing the operation of the continuity evaluation unit when the continuity evaluation algorithm is applied to the element image of the cavity. FIG. 16 is a diagram for explaining a method for evaluating the continuity of two specimen images. FIG. 17 is a diagram illustrating a method for changing the arrangement order of a series of sample images. FIG. 18 is a block diagram illustrating a configuration of an image selection unit in the first modification. FIG. 19 is a diagram for explaining a result when alignment is performed using a specimen image. FIG. 20 is a diagram illustrating a result when alignment is performed using element images.

  Hereinafter, embodiments of an image processing apparatus, a microscope system, an image processing method, and an image processing program according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to these embodiments. Moreover, in description of each drawing, the same code | symbol is attached | subjected and shown to the same part.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of a microscope system according to Embodiment 1 of the present invention. As shown in FIG. 1, a microscope system 1 according to Embodiment 1 includes a microscope apparatus 10 that generates an observation image of a specimen, a slide loader 20 that sequentially supplies a slide on which the specimen is placed, to the microscope apparatus 10. The image acquisition unit 30 captures an observation image generated by the microscope apparatus 10 and acquires a sample image, and the image processing apparatus 40 processes the sample image acquired by the image acquisition unit 30.

FIG. 2 is a schematic diagram schematically showing the configuration of the microscope apparatus 10 and the slide loader 20.
As shown in FIG. 2, the microscope apparatus 10 includes a substantially C-shaped arm unit 100, a sample stage 101 attached to the arm unit 100, an objective lens 102 arranged to face the sample stage 101, It has a trinocular tube unit 103 attached to the arm unit 100, an eyepiece unit 104 provided via the trinocular tube unit 103, and an imaging lens unit 105 connected to the trinocular tube unit 103. An image acquisition unit 30 is provided at the end of the imaging lens unit 105. The arm unit 100 includes, as an illumination system, a transmission illumination light source 106a and a transmission illumination optical system 106b, an epi-illumination light source 107a and an epi-illumination optical system 107b, and a cube that holds a plurality of optical cubes 108a and 108b in a switchable manner. Unit 108 is provided. Such a microscope apparatus 10 is connected to a microscope controller 110 that outputs a control signal to each part of the microscope apparatus 10 under the control of the image processing apparatus 40.

  The specimen stage 101 is provided with a driving device 101a that moves the specimen stage 101 three-dimensionally. The driving device 101a performs focusing by changing the specimen stage 101 in the optical axis direction (Z direction) of the objective lens under the control of the microscope controller 110. The driving device 101a changes the field of view of the objective lens 102 by moving the sample stage 101 in the XY plane under the control of the microscope controller 110.

  The objective lens 102 is attached to a revolver 109 that can hold a plurality of objective lenses (for example, the objective lens 102 ′) having different magnifications. By rotating the revolver 109 and changing the objective lenses 102 and 102 ′ facing the sample stage 101, the magnification of the image in the field of view can be changed.

  The trinocular tube unit 103 branches the observation light of the specimen SP incident from the objective lens 102 into the direction of the image acquisition unit 30 and the direction of the eyepiece unit 104. The eyepiece unit 104 is for the user to directly observe the specimen SP.

  The imaging lens unit 105 is provided with a zoom unit including a plurality of zoom lenses and a drive unit (none of which is shown) that changes the positions of these zoom lenses. The zoom unit adjusts the position of the zoom lens under the control of the microscope controller 110 to enlarge or reduce the imaging target in the imaging field.

  The transmission illumination optical system 106b collects various illumination members (collector lens, filter unit, field stop, shutter, aperture stop, condenser) that collect the transmission illumination light emitted from the transmission illumination light source 106a and guide it in the direction of the observation optical path L. Optical element unit, top lens unit, etc.). On the other hand, the epi-illumination optical system 107b collects various optical members (filter unit, shutter, field stop, aperture stop, etc.) that collect the epi-illumination light emitted from the epi-illumination light source 107a and guide it in the direction of the observation optical path L. Including.

  The cube unit 108 includes a plurality of optical cubes 108a and 108b, and switches the optical cube arranged on the observation optical path L according to various spectroscopic methods such as transmission bright field observation and fluorescence observation.

  The slide loader 20 is a transport device that stores the sample slides SP1, SP2,... On which the sample is placed and transports it between the sample stage 101. A loader control unit 210 that operates under the control of the image processing apparatus 40 is connected to the slide loader 20. The slide loader 20 sequentially supplies specimen slides SP1, SP2,... To be observed in conjunction with the movement of the specimen stage 101 whose position is changed under the control of the microscope controller 110 under the control of the loader control unit 210. .. Collected specimen slides SP1, SP2,... The slide loader 20 is provided with a barcode reader 220. The barcode reader 220 reads a barcode label attached to a specimen slide supplied to the microscope apparatus 10 and transmits a read signal to the image processing apparatus 40. Thereby, in the image processing apparatus 40, the specimen slide being observed by the microscope apparatus 10 is associated with the image data of the specimen image captured by the image acquisition unit 30.

  The image acquisition unit 30 includes an imaging element such as a CCD, for example, and is realized by a multiband camera that can capture color images having pixel levels (pixel values) in a plurality of different wavelength bands (bands) in each pixel. The image acquisition unit 30 receives light (observation light) emitted from the objective lens 102 and incident through the imaging lens unit 105, generates image data corresponding to the observation light, and outputs the image data to the image processing device 40. To do. Although a multiband camera is used in this embodiment, an RGB camera that is generally used without using a multiband camera, and a color measurement sensor that is separate from the RGB camera, It is good also as a structure which obtains a color characteristic (estimates a spectrum and pigment amount) using both the image of a sensor and an RGB camera. For example, it is good also as a structure like the microscope system described in Unexamined-Japanese-Patent No. 2011-99823.

  The image processing apparatus 40 includes an input unit 41 that receives an instruction and information input to the image processing apparatus 40, an image input unit 42 that is an interface that receives an input of a microscope image output from the image acquisition unit 30, a microscope image, A display unit 43 that displays other information, a storage unit 44, a calculation unit 45 that performs predetermined image processing on the microscope image, and a control unit 46 that controls the operations of these units and the operation of the image acquisition unit 30. With.

  The input unit 41 includes input devices such as a keyboard, various buttons, and various switches, and pointing devices such as a mouse and a touch panel. The input unit 41 receives signals input via these devices and inputs them to the control unit 46.

  The display unit 43 is realized by a display device such as an LCD (Liquid Crystal Display), an EL (Electro Luminescence) display, or a CRT (Cathode Ray Tube) display, and displays various screens according to control signals output from the control unit 46. indicate.

  The storage unit 44 includes update-recordable flash memory, RAM, ROM, and other semiconductor memories, a built-in hard disk connected via a data communication terminal, MO, CD-R, DVD-R, and the like, and the recording medium. This is realized by a reading device or the like that reads recorded information. The storage unit 44 stores image data output from the image acquisition unit 30, various programs executed by the calculation unit 45 and the control unit 46, and various setting information. Specifically, the storage unit 44 assigns an accurate arrangement order according to the order in which the original biological tissue specimens are sliced to the specimen images corresponding to the series of slice specimens captured by the image acquisition unit 30. 441 is stored. The storage unit 44 also stores various data used when executing the image processing program 441 (for example, teacher data such as a dye amount distribution in a standard sample subjected to HE staining, information on various dyes, arithmetic expressions, and coefficients). Etc.).

  The calculation unit 45 reads various programs stored in the storage unit 44 into hardware such as a CPU, thereby slicing a biological tissue specimen with respect to a series of specimen images corresponding to the image data stored in the storage unit 44. The image processing for assigning an accurate arrangement order according to the order is performed.

  More specifically, the calculation unit 45 selects an image selection unit 451 that selects a combination of sample images that are estimated to be continuous in a series of sample images as a continuous slice candidate image group, and each sample image. An element image creation unit 452 that extracts at least one type of tissue constituent element from the specimen image based on the color feature amount, creates an element image that is an image of the extracted tissue constituent element for each type, and between the specimen images , Including continuity evaluation algorithms 453a, 453b,... For comparing the corresponding element images and evaluating the continuity, and determining the arrangement order between the sample images based on the continuity evaluation result. And a continuity evaluation unit 453.

  Here, the tissue constituent element is each element constituting the slice specimen, and specifically, a biological tissue such as a cell membrane, a cytoplasm, a cell nucleus, a blood vessel, blood, a cavity (a cross section of a tubular tissue or a gap in the cytoplasm). In addition to the elements constituting the section, it includes the wrinkles and distortion of the section generated when preparing the section specimen, bubbles generated in the slide, dust adhering to the slide, and the like.

  Of the configuration of the calculation unit 45, the continuity evaluation unit 453 determines continuity between sample images according to the continuity evaluation algorithms 453a, 453b,... Created for each tissue component, and based on this continuity. The order of arrangement is determined.

  In addition, the calculation unit 45 performs alignment between the arrangement order providing unit 454 that assigns the arrangement order to the series of sample images and the sample images in which the arrangement order continues based on the determination result by the continuity evaluation unit 453. The alignment unit 455 to perform, a rearrangement unit 456 that rearranges a series of specimen images according to the arrangement order given to each specimen image, and a three-dimensional view from the series of specimen images in which the registration and arrangement order are changed A 3D image constructing unit 457 that constructs a simple image (hereinafter referred to as a 3D image).

  FIG. 3 is a block diagram illustrating an internal configuration of the image selection unit 451. As illustrated in FIG. 3, the image selection unit 451 includes a contour extraction unit 451 a that extracts a contour from an arbitrary sample image of a series of sample images, and the arbitrary sample image and another sample image using the extracted contour. A similarity determination unit 451b that performs similarity determination between the sample images, and an image extraction unit that extracts sample images that are determined to have similar overall shapes to a certain degree based on the result of the similarity determination as a continuous segment candidate image group 451c.

  Further, the element image creation unit 452 is based on a spectrum estimation unit 452a that estimates a color component spectrum (spectral transmittance) at each pixel position based on the pixel value of each pixel in the sample image, and on the basis of the color component spectrum. A pigment amount estimation unit 452b that estimates the pigment amount in the sample portion corresponding to the pixel position, a pigment amount correction unit 452c that corrects the estimated pigment amount based on information on the standard sample, and an estimated or corrected pigment A color feature amount distribution in a color feature amount space having the amount as a component, a class classification unit 452d for classifying the distribution into a class corresponding to the organization component, and a pixel corresponding to the color feature amount belonging to each class A class-specific image creation unit 452e that extracts and creates an image for each class as an element image. The spectrum estimation unit 452a, the pigment amount estimation unit 452b, and the pigment amount correction unit 452c are provided as a color feature amount acquisition unit.

The control unit 46 is realized by reading various programs stored in the storage unit 44 into hardware such as a CPU. Based on various data stored in the storage unit 44 and various information input from the input unit 41, the control unit 46 instructs each unit of the microscope apparatus 10, the slide loader 20, the image acquisition unit 30, and the image processing apparatus 40. By performing the data transfer, the operation of the entire microscope system 1 is comprehensively controlled.
Such an image processing device 40 is realized by a general-purpose device such as a personal computer or a workstation.

Next, the operation of the microscope system 1 will be described. FIG. 4 is a flowchart showing the operation of the microscope system 1.
The microscope system 1 creates a series of section specimens by performing HE staining on each section specimen obtained by slicing a biological tissue specimen, and images a series of these specimen specimens to obtain a series of specimen images. Then, a process of rearranging by assigning an accurate arrangement order according to the order of slicing the original biological tissue specimen is executed.

  First, in step S10, the image acquisition unit 30 changes the field of view of the objective lens 102 by moving the specimen stage 101 within the XY plane of the slice specimen supplied to the microscope apparatus 10 by the slide loader 20, and multiband. The image data of the image (specimen image) corresponding to the entire slice specimen is output to the image processing device 40 by sequentially capturing the images and joining the captured images. The image acquisition unit 30 sequentially performs these operations on a plurality of section specimens to obtain a plurality of specimen images. The image processing apparatus 40 stores the received image data in the storage unit 44. FIG. 5 is a diagram showing an example of the specimen image acquired in this way.

  In the following description, an example in which the present invention is applied to a so-called virtual slide device will be described, but the present invention is not limited to this. For example, for an image obtained by photographing one field of a specimen, the following is described. You may implement the process similar to description.

  In step S <b> 11, the image selection unit 451 selects a continuous section candidate image group from a series of sample images corresponding to the image data stored in the storage unit 44. More specifically, the contour extraction unit 451a extracts an arbitrary sample image, and applies a known contour extraction method such as edge processing to the sample image to extract a contour. Subsequently, the similarity determination unit 451b performs template matching processing on other sample images using the extracted contour as a template. The image extraction unit 451c extracts sample images whose similarity calculated at that time is equal to or greater than a predetermined threshold as a continuous segment candidate image group.

  Note that the contour extraction unit 451a may randomly extract a sample image for contour extraction from a series of sample images or may extract a sample image from a series of sample images at a predetermined interval. Alternatively, when the input unit 41 receives an input signal corresponding to the sample image selected by the user, the contour extraction unit 451a may extract the sample image according to the input signal.

  In subsequent step S12, the element image creation unit 452 creates an element image by extracting at least one type of tissue constituent element for each specimen image (reference: Japanese Patent Application Laid-Open No. 2009-14355).

FIG. 6 is a flowchart showing the detailed operation of the element image creation unit 452.
In step S121, the spectrum estimation unit 452a acquires a pixel value of each band at each pixel position in the sample image, and estimates a color component spectrum at each pixel position by Wiener estimation.

In step S122, the pigment amount estimation unit 452b estimates the pigment amount fixed to the sample portion corresponding to each pixel position based on the spectrum estimated in step S121. Here, the amounts of hematoxylin (hereinafter referred to as dye H) that stained the cell nucleus, eosin that stained cytoplasm (hereinafter referred to as dye E), and eosin that stained red blood cells (hereinafter referred to as dye R) are estimated. . This dye amount solves a simultaneous equation in which the following equation (1) that gives the spectral transmittance t (x, λ) of the wavelength component λ at the pixel position x is simultaneous for a plurality of wavelength components λ according to the Lambert-Beer rule. Is calculated by
In Equation (1), coefficients k _H (λ), k _E (λ), and k _R (λ) are coefficients corresponding to the dye H, the dye E, and the dye R, respectively, and are determined depending on the wavelength. It is a unique value. The value d _H represents the amount of the dye H, the value d _E represents the amount of the dye E, and the value d _R represents the amount of the dye R.

In step S123, the class classification unit 452d creates a pigment amount distribution in the color feature amount space having the pigment amount (d _H , d _E ) estimated in step S122 as a component, and clusters these distributions. FIG. 7 is a diagram illustrating a color feature amount space in which the pigment amount (d _H , d _E ) at each pixel position of the sample image M illustrated in FIG. 5 is plotted.

In step S124, the dye amount correction unit 452c corrects the distribution of the dye amount based on the information related to the standard specimen subjected to HE staining. FIG. 8 is a diagram for explaining a method for correcting the dye amount distribution. First, the dye amount correction unit 452c first calculates the dye amount distribution in an image (hereinafter referred to as a standard image) obtained by imaging a standard specimen subjected to HE staining from the teacher data stored in advance in the storage unit 44. D ₀ and distribution ranges Δd _H and Δd _E of the dye amount distribution D ₀ are acquired. Then, the dye amount correction unit 452c, in order to correspond to the pigment weight distribution D ₁ of the specimen image to be processed distribution range [Delta] d _H and [Delta] d _E, performed shift pigment weight distribution D _1, enlargement, the processing such as reduction . By executing such processing for each specimen image, the color distribution among the plurality of specimen images is homogenized.

  In step S125, the class classification unit 452d clusters the corrected pigment amount distribution. As a clustering method, a known method such as a water separation process, a hierarchy method, a k-means method, an expected value maximization algorithm (EM algorithm), a self-organizing map, or the like can be used. Thereby, clusters a1 to a7 shown in FIG. 7 are obtained.

Furthermore, the class classification unit 452d acquires the barycentric positions of the clusters a1 to a7. Then, by comparing each centroid position with the centroid of the cluster in the dye amount distribution D ₀ of the standard image, the clusters a1 to a7 are classified into classes corresponding to the tissue constituent elements. For example, in the case of FIG. 7, the clusters a3 and a4 are identified as the class c1 corresponding to the cytoplasm because the amount of pigment of eosin is large. A part of the clusters a5, a7 and cluster a2 are identified as class c2 corresponding to the cell nucleus because the amount of hematoxylin pigment is moderately large. Cluster a1 is identified as class c3 corresponding to red blood cells. Cluster a6 is identified as class c4 corresponding to the cavity because the pigment is very thin. As with these examples, it is possible to associate clusters and classes with respect to dust, soot and the like that are captured in the image. For example, since dust, wrinkles, bubbles, and the like are nearly black, hematoxylin is once classified into a cluster a2 that is very dark. By classifying the cluster a2 in more detail, it is possible to specify classes corresponding to dark portions of the cocoon (class c5), dust (class c6), and cell nucleus, respectively.

  Note that the correspondence between each class and organization component based on the class classification result is not limited to one-to-one, and depending on the class classification method, for example, there may be a plurality of classes corresponding to one organization component. These clustering methods may be determined adaptively depending on the tissue component, staining method, and the like.

  In step S126, the class-specific image creation unit 452e creates element images for each class (that is, for each tissue constituent element) by extracting a pixel region corresponding to the amount of pigment classified into each class from the sample image. .

Examples of images obtained by extracting pixels corresponding to the organizational components of each class are shown in FIGS. An element image M-c1 illustrated in FIG. 9 is an image representing the cytoplasm obtained by extracting pixels corresponding to the class c1. An element image M-c2 illustrated in FIG. 10 is an image representing a cell nucleus from which pixels corresponding to the class c2 are extracted. An element image M-c3 illustrated in FIG. 11 is a red blood cell image obtained by extracting pixels corresponding to the class c3. An element image Mc4 shown in FIG. 12 is an image representing a cavity from which pixels corresponding to the class c4 are extracted. An element image M-c5 illustrated in FIG. 13 is an image representing a bag obtained by extracting pixels corresponding to the class c5. An element image M-c6 illustrated in FIG. 14 is an image representing dust from which pixels corresponding to the class c6 are extracted.
Thereafter, the operation returns to the main routine.

  In step S13, the continuity evaluation unit 453 evaluates continuity between a plurality of sample images using element images of a predetermined class. This evaluation is performed between the sample images determined as the continuous section candidates in step S11.

  Here, each tissue component extracted from the specimen image has characteristics according to the characteristics of each type of tissue component, the site of the original biological tissue specimen, the dye that stains the section specimen, etc. ing. For example, since cell nuclei and red blood cells are very small in size, the same cross-sectional shape rarely appears between adjacent section samples. On the other hand, since the sizes of cytoplasm, blood vessels, cavities, etc. are relatively large, a similar cross-sectional shape often appears between adjacent section samples. In addition, since wrinkles and dust are individually generated when preparing a section sample, it cannot be used for evaluation of continuity between adjacent section samples.

  The continuity evaluation algorithms 453a, 453b,... Held by the continuity evaluation unit 453 are information known in advance such as characteristics for each type of tissue component, characteristics of a part of a biological tissue specimen, and characteristics of a pigment ( These are also referred to as foresight information).

  Hereinafter, as an example, a continuity evaluation algorithm created for an element image of a cavity will be described. FIG. 15 is a flowchart showing the operation of the continuity evaluation unit 453 when the continuity evaluation algorithm is applied to the element image of the cavity.

  First, in step S131, the continuity evaluation unit 453 generates cavity element images M1-c4, M2-c4, M3-c4... From the element images created in common for the sample images M1, M2, M3. Select (see FIG. 16).

  In step S132, the continuity evaluation unit 453 deletes the small cavities P1, P2, P3,... From the element images M1-c4, M2-c4, M3-c4.

  Subsequently, the continuity evaluation unit 453 performs steps S133 to S135 for each of the element images M1-c4, M2-c4, M3-c4. First, in step S133, the continuity evaluation unit 453 picks up one of the element images M1-c4, M2-c4, M3-c4. In subsequent step S134, the continuity evaluation unit 453 calculates the similarity between the picked-up element image and all other element images using a known method such as pattern matching. In step S135, the calculated similarity is stored in the storage unit 44.

In step S136, the continuity evaluation unit 453 comprehensively evaluates all the calculated similarities so that the similarity between the element images M1-c4, M2-c4, M3-c4. Detect the sort order. In that case, the similarity between images is calculated and rearranged in all image combinations, or two pair images are first set, the similarity between the pair images is calculated, and the rearrangement is repeated. A known sorting method is used. For example, the element images M1-c4, M2-c4, M3-c4... May be sorted using a method such as a similarity map or a composite sort method.
Thereafter, the operation returns to the main routine.

  In step S14, the arrangement order assigning unit 454 assigns the arrangement order of the element images M1-c1, M2-c2, M3-c2... Detected in step S16 to the sample images M1, M2, M3.

  In step S15, the rearrangement unit 456 rearranges the sample images M1, M2,... According to the arrangement order assigned in step S14, as shown in FIG. For example, in FIG. 17, the sample image M2 and the sample image M4 are interchanged.

  In step S16, the alignment unit 455 aligns the sample images that are adjacent in the arrangement order (for example, sample images M1 and M4, M4 and M3, M3 and M2,...), And the element image M1 selected in step S13. -C4, M2-c4, M3-c4...

  In step S17, the 3D image constructing unit 457 generates a 3D image in which the sample image aligned in step S16 is three-dimensionally constructed.

  In step S18, the calculation unit 45 outputs the arrangement order information and alignment information of the specimen images M1, M2,... And the generated 3D image. In response to this, the control unit 46 stores the output information in the storage unit 44 and displays a 3D image on the display unit.

  As described above, according to the first embodiment, the continuity between the sample images is evaluated using the element image extracted based on the color feature amount of the sample image and selected based on the foresight information. Based on the evaluation result, the arrangement order of the sample images is determined. Therefore, it is possible to give an accurate arrangement order to each specimen image without being affected by the wrinkles and distortion of the sections generated during the preparation of the section specimens or the influence of unnecessary objects such as bubbles and dust. Therefore, even if the order of the sequence of section specimens or the order of imaging of the section specimens is confused, it is possible to arrange the series of specimen images in an accurate order according to the order in which the original biological tissue specimen is sliced. It becomes possible.

  In addition, according to the first embodiment, the alignment between the specimen images is performed based on the element image selected based on the foresight information. Accurate alignment can be performed without being affected by unnecessary objects such as dust and dirt. Therefore, it is possible to construct a 3D image that accurately reflects the shape and the like of the original biological tissue specimen using the series of specimen images aligned in this way.

In the first embodiment, the continuity evaluation unit 453 evaluates the continuity between the consecutive segment candidate images determined to be somewhat similar to each other by the image selection unit 451, so that the amount of calculation can be reduced. .
Note that the image selection unit 451 is not necessarily provided, and when the image selection unit 451 is not provided, the continuity evaluation unit 453 may evaluate the continuity for all combinations of sample images in a series of sample images. Good.

  In the first embodiment, the element image is created based on the distribution of the pigment amount homogenized by the pigment amount correction unit. Therefore, even when the preparation conditions and imaging conditions of the section specimen are different (when the creator and facility of the section specimen are different, when the manufacturer of the staining dye is different, when the model number of each device constituting the imaging system 1 is different, etc.) , Variation in results can be suppressed.

In the first embodiment, the continuity of the specimen image is evaluated using the elemental image of the cavity. In addition to the cavity, a plurality of sections such as a cell membrane, cytoplasm, blood vessel, and mammary gland are used. The continuity of the specimen image may be evaluated using an element image of an existing tissue component.
Further, continuity may be evaluated comprehensively using images of a plurality of tissue components.

(Modification 1)
Next, Modification 1 of Embodiment 1 will be described.
FIG. 18 is a block diagram illustrating another configuration example of the image selection unit 451 illustrated in FIG. As shown in FIG. 18, an image selection unit 471 that is a modification of the image selection unit 451 performs a binarization process on each of the series of sample images, thereby extracting a background, and a background extraction unit 471a. A similarity determination unit 471b that performs similarity determination between a plurality of sample images based on the extracted background, and an image that extracts a plurality of sample images determined to be somewhat similar to each other by the similarity determination unit 471b as a continuous segment candidate image group And an extraction unit 471c. In this case, the continuity evaluation unit 453 evaluates continuity between the continuous segment candidate images extracted by the image extraction unit 471c.

(Modification 2)
Next, a second modification of the first embodiment will be described.
The image processing device 40 may include an output device that prints out a barcode label corresponding to the sample image after the arrangement order is changed. In this case, it is possible to appropriately manage the specimen slides SP1, SP2,... By attaching the output barcode labels to the specimen slides SP1, SP2,. It becomes.

(Modification 3)
Next, a third modification of the first embodiment will be described.
In the first embodiment, continuity evaluation and alignment between specimen images are performed using predetermined element images. However, the continuity evaluation unit 453 may evaluate continuity by directly using an image after removing tissue components such as dust and wrinkles from the sample image (hereinafter referred to as a removed sample image). Similarly, the alignment unit 455 may perform alignment using the removed specimen image directly. Further, the 3D image construction unit 457 may construct a 3D image from the removed specimen image.

(Embodiment 2)
Next, a second embodiment of the present invention will be described.
In the second embodiment, the element image created by the element image creation unit 452 may be used for alignment between two specimen images extracted from a series of specimen images.

Here, as shown in FIG. 19, consider a case where the sample images M _A and M _B are aligned with each other. These specimen image M _A, the M _B, various tissue components (e.g., cavities ma1 and dust ma2 on the specimen image M _A, lumen mb1 on specimen image M _B) is captured. Such specimen image M _A, when combined directly position M _B, tissue components (e.g., dirt ma2) depending on the shape or luminance, different tissue components with each other (e.g., dust mb1 against lumen ma2) mutually It may be aligned. In this case, the aligned sample image M _A, to constitute 3D image using M _B, it becomes impossible to reproduce the three-dimensional shape of the original biological tissue specimen. Also, when two specimen images are aligned and overlapped or arranged for comparison diagnosis, the display is shifted and the difference between the two images cannot be displayed clearly. End up.

Therefore, in the second embodiment, as shown in FIG. 20, first, from the sample images M _A and M _B , element images M _A -c 1 and M _B − of tissue constituent elements (cavities ma 1 and mb 1) corresponding to each other. c1 is extracted, and alignment is performed between the element images M _A -c1 and M _B -c1. The element image used at this time may be determined based on the foresight information. Then, the alignment elements already image M _A -c1, based on the M _B -c1, specimen image M _A, to determine the relative position of M _B.

  According to the second embodiment described above, highly accurate alignment can be performed on any two specimen images. Therefore, when a 3D image is formed from a series of specimen images that are aligned with each other, an output image that accurately reflects the three-dimensional shape of the original biological tissue specimen can be obtained. In addition, when two specimen images are aligned and overlapped or displayed for comparison diagnosis, accurate alignment is performed and the difference between the two images is displayed in an easy-to-understand manner. It becomes possible to do.

  In the first and second embodiments described above, the image processing for a plurality of specimen images obtained by sequentially imaging a plurality of section specimens obtained by slicing one biological tissue specimen into a plurality of sections has been described. However, Embodiments 1 and 2 and Modifications 1 to 3 can also be applied to image processing for a plurality of cross-sectional images captured while changing the focal point of a thick specimen. Even in this case, the element image created based on the color feature (pigment amount) distribution calculated from each cross-sectional image is used to determine the continuity between the cross-sectional images and determine the order of alignment, It can be carried out. In such a case, the sample may be imaged while automatically changing the focus in the microscope system, or the sample may be imaged while the user manually changes the focus. In particular, when taking an image while changing the focus manually, the focus setting order is wrong, or a certain focus position is missed, so an additional image is taken later. Sometimes, imaging is performed while changing the focus more finely. In such a case, a situation may occur in which the order of the acquired images does not correspond to a change in the focal position according to a certain rule. In addition, the sample position may be slightly shifted during imaging. However, according to the present embodiment, it is possible to automatically determine an accurate arrangement order of a plurality of images even in the above-described case, and it is possible to further save the trouble of imaging.

  Embodiments 1 and 2 and Modifications 1 to 3 described above are not limited as they are, and various inventions can be made by appropriately combining a plurality of components disclosed in the embodiments and modifications. Can be formed. For example, some components may be excluded from all the components shown in the embodiment. Or you may form combining the component shown in different embodiment suitably.

(Appendix 1)
An image processing apparatus that processes a plurality of images corresponding to a plurality of slice specimens obtained by staining a slice of a biological tissue specimen with a predetermined dye,
For each image of the plurality of images, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and an element image that is an image in which the tissue component is reflected for each type is obtained. An element image creation section to be created;
An image processing apparatus comprising: an alignment unit configured to perform alignment between the plurality of images based on the position of the tissue constituent element in the element image.
(Appendix 2)
The element image creation unit
A color feature amount acquisition unit for acquiring a color feature amount of a pixel in each image;
Creating a distribution of the color feature quantity, and classifying the distribution into at least one class corresponding to the at least one type of tissue component;
Extracting pixels corresponding to the color feature amounts belonging to each class, and creating an image for each class as the element image,
The image processing apparatus according to appendix 1, characterized by comprising:
(Appendix 3)
The color feature amount is a pigment amount of the pigment in a sample portion corresponding to each pixel position in the image,
The color feature amount acquisition unit
A spectrum estimation unit that estimates a spectrum of a color component at each pixel position based on a pixel value of each pixel in each image;
A pigment amount estimation unit that estimates the pigment amount in the sample portion based on the spectrum of the color component;
Including
The image processing apparatus according to appendix 2, wherein the class classification unit classifies the pigment amount distribution by clustering the pigment amount distribution in a color feature amount space including the pigment amount as a component. .
(Appendix 4)
The color feature amount acquisition unit further includes a dye amount correction unit that homogenizes the distribution of the dye amount among the plurality of images,
The image processing apparatus according to appendix 3, wherein the class classification unit classifies the distribution of the dye amount corrected by the dye amount correction unit.
(Appendix 5)
The image processing apparatus according to any one of appendices 1 to 4, wherein the at least one type of tissue constituent element includes any one of a cell membrane, a cytoplasm, a cell nucleus, a blood vessel, blood, and a cavity.
(Appendix 6)
The image processing apparatus according to any one of appendices 1 to 5,
A microscope apparatus that generates observation images of a plurality of slice specimens obtained by staining a slice of a biological tissue specimen with a predetermined dye; and
An image acquisition unit for acquiring a digital image corresponding to the observed image;
A microscope system comprising:
(Appendix 7)
The microscope system according to appendix 6, further comprising a slide transport device that supplies a plurality of slides each having the plurality of section specimens mounted thereon to the microscope device.
(Appendix 8)
An image processing method for processing a plurality of images corresponding to a plurality of slice specimens obtained by staining a slice of a biological tissue specimen with a predetermined dye,
For each image of the plurality of images, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and an element image that is an image in which the tissue component is reflected for each type is obtained. An element image creation step to be created;
And an alignment step of performing alignment between the plurality of images based on the position of the tissue component in the element image.
(Appendix 9)
An image processing program for processing a plurality of images corresponding to a plurality of slice specimens obtained by staining a slice obtained by slicing a biological tissue specimen with a predetermined dye,
For each image of the plurality of images, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and an element image that is an image in which the tissue component is reflected for each type is obtained. An element image creation step to be created;
An image processing program that causes a computer to execute an alignment step of performing alignment between the plurality of images based on the position of the tissue component in the element image.

DESCRIPTION OF SYMBOLS 1 Microscope system 10 Microscope apparatus 100 Arm part 101 Sample stage 101a Drive apparatus 102,102 'Objective lens 103 Trinocular tube unit 104 Eyepiece unit 105 Imaging lens unit 106a Light source for transmitted illumination 106b Transmitted illumination optical system 107a Light source for incident illumination 107b Epi-illumination optical system 108 Cube unit 108a, 108b Optical cube 109 Revolver 110 Microscope controller 20 Slide loader 210 Loader control unit 220 Bar code reader 30 Image acquisition unit 40 Image processing device 41 Input unit 42 Image input unit 43 Display unit 44 Storage unit 45 arithmetic unit 441 image processing program 451 image selection unit 451a contour extraction unit 451b similarity determination unit 451c image extraction unit 452 element image creation unit 452a Couttle estimation unit 452b Dye amount estimation unit 452c Dye amount correction unit 452d Class classification unit 452e Class-specific image creation unit 453 Continuity evaluation unit 453a, 453b,... 457 3D image construction unit 46 control unit 471 image selection unit 471a background extraction unit 471b similarity determination unit 471c image extraction unit

Claims (15)

For a plurality of images corresponding to a plurality of slice samples obtained by slicing and staining a biological tissue sample, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and for each type An element image creation unit for creating an element image that is an image of
Arrangement order determination that includes a continuity evaluation unit that evaluates the continuity of the same type of element images between the plurality of images, and that determines the arrangement order of the plurality of images based on the continuity evaluation result And
An image processing apparatus comprising: An image selection unit that selects a plurality of images that are estimated to be continuous from the plurality of images as a continuous section candidate image group;
The image processing apparatus according to claim 1, wherein the arrangement order determination unit determines an arrangement order between the consecutive segment candidate image groups. The image selection unit
A contour extraction unit that extracts a contour from the plurality of images;
A similarity determination unit that performs similarity determination between an arbitrary image of the plurality of images and another image based on the contour;
An image extraction unit that extracts images determined to be similar to each other by the similarity determination unit as the continuous segment candidate image group;
The image processing apparatus according to claim 2, further comprising: The image selection unit
A background extraction unit that extracts a background from each of the plurality of images;
A similarity determination unit that performs similarity determination between an arbitrary image of the plurality of images and another image based on the background;
An image extraction unit that extracts images determined to be similar to each other by the similarity determination unit as the continuous segment candidate image group;
The image processing apparatus according to claim 2, further comprising: The element image creation unit
A color feature amount acquisition unit for acquiring a color feature amount of a pixel in each image;
Creating a distribution of the color feature quantity, and classifying the distribution into at least one class corresponding to the at least one type of tissue component;
Extracting pixels corresponding to the color feature amounts belonging to each class, and creating an image for each class as the element image,
The image processing apparatus according to claim 1, wherein the image processing apparatus includes: The color feature amount is a pigment amount of the pigment in a sample portion corresponding to each pixel position in the image,
The color feature amount acquisition unit
A spectrum estimation unit that estimates a spectrum of a color component at each pixel position based on a pixel value of each pixel in each image;
A pigment amount estimation unit that estimates the pigment amount in the sample portion based on the spectrum of the color component;
Including
6. The image processing according to claim 5, wherein the class classification unit classifies the pigment amount distribution by clustering the pigment amount distribution in a color feature amount space including the pigment amount as a component. apparatus. The color feature amount acquisition unit further includes a dye amount correction unit that homogenizes the distribution of the dye amount among the plurality of images,
The image processing apparatus according to claim 6, wherein the class classification unit classifies the distribution of the dye amount corrected by the dye amount correction unit.   The said continuity evaluation part evaluates the continuity of the said element images by applying the continuity evaluation algorithm defined for every said structure | tissue component, The any one of Claims 1-7 characterized by the above-mentioned. The image processing apparatus according to item 1.   The continuity evaluation unit performs the continuity evaluation based on at least one of a characteristic of the tissue component, a part of the biological tissue specimen, and the predetermined dye. Item 9. The image processing apparatus according to Item 8. The at least one tissue component includes at least one of a cell membrane, cytoplasm, cell nucleus, blood vessel, blood, and cavity;
9. The continuity evaluation unit evaluates the continuity based on an element image corresponding to any one of the cell membrane, the cell nucleus, the blood, and the cavity. Or the image processing apparatus according to 9.   The image according to claim 1, further comprising a three-dimensional image configuration unit configured to three-dimensionally configure the plurality of images based on a determination result by the arrangement order determination unit. Processing equipment.   The image processing according to any one of claims 1 to 11, further comprising an alignment unit configured to perform alignment between the plurality of images based on the position of the tissue component in the element image. apparatus. The image processing apparatus according to any one of claims 1 to 12,
A microscope apparatus that generates observation images of a plurality of slice specimens obtained by staining a slice of a biological tissue specimen with a predetermined dye; and
An image acquisition unit for acquiring a digital image corresponding to the observed image;
A microscope system comprising: For a plurality of images corresponding to a plurality of slice samples obtained by slicing and staining a biological tissue sample, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and for each type An element image creation step of creating an element image that is an image of
An arrangement order determination step for evaluating the continuity of the same kind of element images between the plurality of images and determining the arrangement order of the plurality of images based on the evaluation result of the continuity;
An image processing method comprising: For a plurality of images corresponding to a plurality of slice samples obtained by slicing and staining a biological tissue sample, at least one type of tissue component is extracted based on the color feature amount of each pixel in the image, and for each type An element image creation step of creating an element image that is an image of
An arrangement order determination step for evaluating the continuity of the same kind of element images between the plurality of images and determining the arrangement order of the plurality of images based on the evaluation result of the continuity;
An image processing program for causing a computer to execute.