NL2018386B1

NL2018386B1 - Structured illumination scanning microscopy

Info

Publication number: NL2018386B1
Application number: NL2018386A
Authority: NL
Inventors: Manders Erik; Breedijk Ronald
Original assignee: Univ Amsterdam
Priority date: 2017-02-16
Filing date: 2017-02-16
Publication date: 2018-09-06
Also published as: WO2018151599A1; EP3583459A1

Abstract

A method of forming a high-resolution image of a sample using a scanning microscope controlled by a processor is described where in the method comprises the steps of receiving or generating control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters, preferably the one or more pattern parameters, including at least one of: a spatial frequency, a periodicity direction, an initial phase; using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal; controlling an imaging system to capture multiple images, each image being associated with an emission light signal of one of the multiple exposures, and, using a structured light reconstruction algorithm for forming a high-resolution image on the basis of the one or more pattern parameters.

Description

Field of the invention

The invention relates to structured illumination microscopy, and, in particular, though not exclusively, to methods and systems for structured illumination scanning microscopy and a computer program product for executing such methods .

Background of the invention

Super resolution microscopy methods are known for resolving spatial resolutions beyond the diffraction limit. An example of such a super-resolution method is the well-known structured illumination microscopy (SIM) as e.g. described in the article by Gustafsson et al, Journal of Microscopy, Vol. 198, Pt 2, May 2000, pp. 82-87. Herein, a sample is illuminated with a series of light patterns (also referred to as structured light) comprising bright maxima (associated with high light intensities) and dark minima (associated with low light intensities), which cause normally inaccessible highresolution information to be encoded into observed images. Recorded images are subsequently processed to extract the high-resolution information and construct a super-resolution image, i.e. an image having a resolution surpassing the diffraction limit.

The goal of SIM is to obtain a super-resolution image of an unknown sample structure - or more precisely - of an unknown spatial distribution of fluorescent dyes. Structured illumination light, typically generated using a diffraction grating, is incident on the spatial distribution of fluorescent dyes and causes the fluorescent dyes to emit emission light. The intensity of emission light originating from a point on the sample is proportional to the product of dye concentration at this point and the effective light intensity of the structured light at this point. The resulting emission light is thus a product of two patterns, namely the structured light and the light of an (unknown) distribution of fluorescent molecules. As a result, the emission light will contain observable Moiré effects from which normally inaccessible high-resolution information can be derived as explained in Gustafsson et al.

The light pattern typically includes a two or three dimensional arrangement of parallel lines in one or more planes perpendicular to the optical axes. This light pattern is used to expose a sample several times, wherein after each exposure, the light pattern is shifted and/or rotated. Recoded images of the emission light of each exposure are used to reconstruct a high resolution image using a known mathematical image reconstruction algorithm. An example of such method is described in Lal et. al, IEEE Journal of selected topics in quantum electronics, Vol. X, No. X, December 2015. For a successful reconstruction of a high-resolution image, it is important that the parameters of the structured light pattern on the sample can be determined as accurate as possible. Deviations from the exact position will cause artefacts in the reconstructed image.

For example a 2D arrangement of parallel lines may be represented by a sinusoid having an amplitude, a modulation depth, an initial phase, a direction and a spatial frequency, wherein the amplitude relates to the light intensity of the bright maxima and dark minima of a light pattern and the modulation depth relates to the difference in light intensities. The spatial frequency defines the number of bright maxima per distance perpendicular to the linedirection. Furthermore, the initial phase defines the position of a light patterns on a sample relative to each other. For reconstructing a high-resolution image it is important to know the parameters of the structured light pattern as accurate as possible .

For structured light generated on the basis of a diffraction pattern it is difficult to control the parameters that define the pattern on the sample. Therefore, the parameters such as the phase information and the modulation depth, is typically retrieved a posteriori from the recorded images using image analysis algorithms. These algorithms can only retrieve these parameters if the recorded images comprise a clear enough footprint of the used structured light pattern. This means that the recorded images must comprise discernible bright maxima and dark minima corresponding with the maxima and minima of the used structured light pattern otherwise the algorithms cannot retrieve the positions of the bright maxima on the sample.

Emission light passes through at least one NAlimiting optical element (e.g. an objective lens) before being incident on an image sensor. However, it is known that such NA-limiting element acts as a spatial low pass filter in the sense that it attenuates the amplitudes of high spatial frequency components. Hence, if the spatial frequency of an applied structured light pattern is too high, i.e. if the structured light pattern is too fine, the amplitude associated with this frequency may be attenuated significantly by the lens system. As a result, the bright maxima and dark minima corresponding to the structured light pattern may only be weakly present in a recorded image. The footprint of the structured light pattern may then not be discernible anymore by the image processing algorithms, which may impede determination of the parameters such as the phase information and the modulation depth. The lens system thus imposes an upper limit to the spatial frequency that can be used for structured light patterns. This is disadvantageous, since the spatial frequency of the applied structured light patterns are preferably as high as possible, because finer structured light patterns yield higher resolutions.

US 2013/0314717 Al discloses a SIM system that includes a modulator to modulate the light such that the sample is exposed to structured light. In this scheme, a modulated focal spot of light is scanned over the sample. During scanning, the modulation is controlled such that the sample is exposed to a 2D pattern of parallel lines wherein the lines make an angle with the scanning direction.

In the proposed system the scanning movement of the focal spot over the sample must be accurately timed with respect the modulation in order to generate an accurate 2D pattern of parallel lines of a high spatial frequency. Further, also in this document the parameters of the light pattern are derived from the captured images.

Hence, there is a need in the art for improved structured light microscopy systems. In particular, there is a need in the art for systems and methods for SIM which allow the use of high frequency light patterns.

Summary of the invention

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a circuit, module or system. Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the C programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks .

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It is an object of the present disclosure to provide methods and systems for structured illumination microscopy using a scanning microscope. In particular, it is an object of the present disclosure to provide methods and systems for scanning structured illumination microscopy (SSIM) that enable the use of periodic light patterns having spatial frequencies that are equal or higher than the spatial frequencies and that eliminate the use of posteriori image analysis for determining the light pattern parameters which are used by the reconstruction algorithm.

In an aspect, the invention relates to a method of forming a high-resolution image of a sample using a scanning microscope controlled by a processor. The method may include: the processor receiving or generating control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters; the processor using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal; the processor controlling an imaging system to capture multiple images, each image being associated with an emission light signal of one of the multiple exposures, and, the processor using an image reconstruction algorithm for forming a highresolution image on the basis of at least one of the one or more pattern parameters and the captured images. In an embodiment, the one or more pattern parameters may include at least one of: a spatial frequency, a periodicity direction, an initial phase, an illumination intensity.

Hence, the method according to the invention uses control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters which is used by a processor, e.g. a computer, of the scanning microscope to expose a sample to multiple different period illumination patterns, i.e. a light pattern that provides a certain light distribution, e.g. a 2D light distribution or a 3D light distribution, within the sample.

As the scanning microscope can accurately control the position of a focused illumination light spot on the sample, a periodic 2D or 3D illumination pattern can be accurately written into the sample using the control information. Here, the focused illumination light spot may comprise a light intensity profile, for example a Gaussian intensity profile, so that areas of the pattern may be understood to have been illuminated with a light intensity exceeding a predetermined threshold light intensity. In the context of this application, the intensity of light may relate to the physical quantity irradiance .

The parameters defining the periodic patterns - which are determined before exposing the sample - may be used in the reconstruction algorithm. The invention thus eliminates or at least substantially reduces the need for a posteriori image analysis in order to determine illumination light pattern parameters that are used by the reconstruction algorithm. This way, light patterns may be used that have a higher spatial frequency than the light patterns that are used in SIM systems known in the prior art.

In an embodiment, controlling the scanning microscope may include controlling a scanning mirror of the scanning microscope to move at least one focused illumination light spot through the sample positioned a sample holder. In an embodiment, the position of the sample and the light spot may be determined on the basis of a reference position associated with the sample or the sample holder. Hence, a reference position associated with the sample (e.g. on the sample) or the sample holder may be used by the scanning microscope in order to determine the position of the focused illumination light spot in the sample. This way, the positions and the orientation of the different illumination patterns that are written into the sample are accurately known and directly correspond the pattern parameters that are used by the processor of the scanning microscope to write the illumination patterns into the sample. These pattern parameters are used by reconstruction algorithm in order to determine a highresolution image. This obviates the need to retrieve information of the illumination pattern a posteriori from the recorded images using image analysis techniques.

In an embodiment, exposing the sample to multiple illumination patterns may include moving the position of a focused illumination light spot of the scanning microscope in accordance with each of said multiple periodic patterns.

In an embodiment, a reference position associated with the sample may be used to define a coordinate system, which may be used by the processor to move the illumination light spot to predetermined positions relative to the reference position. The relative position of the different illumination patterns with respect to the reference position are determined on the basis of a calibration of an optical system, wherein the calibration links settings of the optical system, such as respective positions of said scanning mirror, to positions of the illumination light spot on the sample. Hence, a periodic pattern associated with certain pattern parameters may be defined as a set of coordinates on the sample, which the processor may translate into a set of mirror positions for moving the focused illumination light spot in accordance with the pattern.

In an embodiment, the illumination light spot may comprise a predetermined light intensity pattern comprising at least two spatially arranged light intensity maxima. In an embodiment, the spatially arranged light intensity maxima may define a spatial frequency of the illumination pattern. In an embodiment, the at least two light intensity maxima may be arranged next to each other in a plane perpendicular to the optical axis of the microscope. In another embodiment, the at least two light intensity maxima may be arranged in an axial direction parallel to optical axis of the microscope.

In an embodiment, the predetermined light intensity pattern may comprise at least a 2D arrangement of two or more light intensity maxima; and/or, wherein the predetermined light intensity pattern comprises a 3D arrangement of multiple light intensity maxima.

In an embodiment, the illumination light spot may be a shaped illumination light spot, the shaped light spot comprising at least a first width and at least a second width small than the first width. In an embodiment, during writing, the second width may be arranged perpendicular to the direction in which the focused illumination light spot is moving.

In an embodiment, capturing multiple images may include controlling a rescanning mirror of the scanning microscope to reflect the emission light signal towards an imaging system and to move a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system. The optical response may comprise emission light that is emitted by fluorophores in the sample. The illumination light spot may excite the fluorophores and, as a result, the fluorophores may emit the emission light. The optical response may comprise a collimated emission light beam that is formed using an optical system, such as a lens system.

In an embodiment, controlling the scanning and the rescanning mirror may include rotating the scanning mirror and the rescanning mirror back and forth over a predetermined first angular amplitude and a second angular amplitude respectively. In an embodiment, the ratio between the first and second angular amplitude may be unequal to one. In an embodiment, the second angular amplitude may be selected two times the first angular amplitude.

In an embodiment, forming a high-resolution image further includes: determining a Fourier transform of the different multiple illumination patterns on the basis of the one or more pattern parameters.

In an embodiment, the scanning microscope may comprise at least a pinhole for filtering out-of-focus light out of the illumination light signal. In an embodiment, the scanning microscope may comprise one or more optical elements configured to project an illumination light spot in a plane of interest of the sample onto a confocal conjugate plane of the plane of interest. The use of a pinhole may significantly increase the signal-to-noise ratio of the recorded image data. This is because the sample is exposed to a illumination pattern by scanning an illumination light spot over the sample. This way, a scanning structure illumination microscope may be realized that has confocal characteristics.

In an embodiment, the control information may be adapted to control one or more actuators for controlling a scanning mirror and/or rescanning mirror in order to move a focused illumination light spot in a lateral direction in the sample; and/or, to control one or more optical elements, preferably one or more lenses, in order to move the focused illumination light spot in an axial direction in the sample.

In an aspect, the invention may relate to a scanning microscopy system for forming a high-resolution image of a sample comprising: a computer adapted to control the scanning microscope, the computer comprising a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: receiving or generating control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters, preferably the one or more pattern parameters, including at least one of: a spatial frequency, a periodicity direction, an initial phase; using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal; controlling an imaging system to capture multiple images, each image being associated with an emission light signal of one of the multiple exposures, and, using an image reconstruction algorithm for forming a high-resolution image on the basis of at least one of the one or more pattern parameters and the captured images .

In an embodiment, the executable operations may further comprise: controlling a rescanning mirror of the scanning microscope to reflect the emission light signal towards an imaging system and to move a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system; preferably controlling the scanning and the rescanning mirror includes rotating the scanning mirror and the rescanning mirror back and forth over a predetermined first angular amplitude and a second angular amplitude respectively; more preferably the ratio between the first and second angular amplitude unequal to one; even more preferably, the second angular amplitude being selected two times the first angular amplitude.

In a further aspect, the invention may relate to a control module for controlling a scanning microscopy system comprising: a computer adapted to control the scanning microscope, the computer comprising a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: receiving or generating control information for controlling a scanning microscope, the control information defining a plurality of different periodic patterns on the basis of one or more pattern parameters, preferably the one or more pattern parameters, including at least one of: a spatial frequency, a periodicity direction, an initial phase; using the control information to control the scanning microscope to expose the sample to multiple illumination patterns, each exposure to an illumination pattern causing one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal; controlling an imaging system to capture multiple images, each image being associated with an emission light signal of one of the multiple exposures, and, using an image reconstruction algorithm for forming a high-resolution image on the basis of at least one of the one or more pattern parameters and the captured images.

The invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.

The invention will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments.

Brief description of the drawings

Fig. 1 depicts a scanning microscopy system according to an embodiment of the invention.

Fig. 2A-2C depict a process of illuminating a sample with an illumination pattern using a scanning microscopy system according to an embodiment of the invention.

Fig. 3 depicts illumination patterns for use with a scanning microscopy system according to an embodiment of the invention .

Fig. 4 depicts the reconstruction of an image using a structured illumination microscopy technique.

Fig. 5A-5C depicts various illumination light spots for use in a scanning microscopy system according an embodiment of the invention.

Fig. 6 depicts a scanning microscopy system according to another embodiment of the invention.

Detailed description

Fig. 1 depicts a scanning microscopy system according to an embodiment of the invention. The system may comprise a light source 100 for generating illumination light that is directed onto a sample 112 located on a sample holder (not shown). Illumination light 122 may be directed onto a sample using a first optical system 130, which may include one or more refractive and/or reflective optical elements. For example, in an embodiment, the first optical system may include one or more collimating and/or focusing lenses 102,110, mirrors 104, a set of orthogonal scanning mirrors, schematically depicted by mirror 108 and/or dichroic mirrors 106.

In particular, a lens 102 may be used in order to form a collimated beam of illumination light 122 which may be reflected via a set of mirrors 104,106,108 towards a lens 110 in order to form a focused illumination light spot 126 on the sample. The focused illumination light spot may excite local optical excitations, which generate emission light 124. The emission light may pass through lens 110 and may be directed via a second optical system 132 comprising one or more refractive and/or reflective optical elements to an imaging system 118.

The second optical system may include a lens 110 for forming a collimated beam of the emission light that is reflected by a first scanning mirror 108. The beam of emission light may pass one or more optical elements, e.g. dichroic mirror 106, and reflected by a further second scanning mirror 114, which directs the emission light beams to a focussing lens 116. The second scanning mirror may also be referred to as a rescanning mirror. In an embodiment, the first and/or second scanning mirror may be implemented by a set of orthogonal scanning mirrors.

The focussing lens may focus the emission light beam into focussed emission light spot 128 onto the imaging plane of imaging system 118. Here, the imaging system may include one or more image sensors, e.g. one or more CMOS image sensors or one or more CCDs. Dichroic mirror 106 may be configured such that it functions as a reflector for illumination light 122 and such that it is transparent for emission light 124 (which typically is of a longer wavelength than illumination light 122) .

The first scanning mirror 108 which reflects the illumination light towards the sample and the emission light towards the further the second scanning mirror 114 may be configured as a rotatable mirror which may be controlled by a computer system 120. The computer system may control the actuators of the scanning mirror so that the light spot can be moved in a 2D plane of the sample. Further, the computer system may control the optics of the system in in order to set a focussing height within the sample. By controlling the focus height different 2D planes in the sample at different focus heights can be illuminated with light patterns. This way, the focused illumination light spot 126 can be moved through the sample 112 in accordance with a predetermined 2D or 3D pattern .

For example, when writing a 2D periodic line pattern in the sample, the scanning mirror 108 may be configured to rotate back and forth over a first angular amplitude Al. This way, the mirror can move (scan) the focused illumination light spot through sample causing continuous local optical illumination in the scanning direction of the light spot. Controlling the mirror in three dimensions and controlling the focus of the light spot thus allows writing a 2D or 3D light structure in the sample.

The computer may comprise one or more processor connected to a memory comprising computer code that, when executed by the one or more processor, may transform a predetermined pattern into control information for the optics (the mirrors and the lenses) so that light pattern can be written in the sample.

In an embodiment, the pattern may be defined by a mathematical function including a number of parameters (e.g. frequency, initial phase, amplitude, modulation depth, etc.) which can be easily transformed into control information. By executing the computer code, the control information may control the actuators and optical elements of the system in order to write the pattern into the sample.

In an embodiment, the one or more processors connected to the memory comprising the computer code may be integrated in a structured light microscopy (SIM) module 132 that is configured to transform a predetermined 2D or 3D pattern in to control information associated with a desired illumination pattern that is used in the SIM process.

Illumination patterns can be designed in advance using e.g. a software application that may include a drawing program for designing a desired light pattern, wherein the defined pattern represents an exact copy of the illumination light pattern that is written into the sample.

During scanning, emission light originating from the moving illumination light spot is formed into a collimated beam 124 of emission light. The collimated beam may be reflected via the scanning mirror 108 towards the second scanning mirror 114, a rotatably mounted scanning mirror, which is controlled to reflect the light beam towards the focusing lens 116 in order form a moving emission light spot 128 onto the imaging plane of the imaging system 118.

The second scanning mirror 114 may be configured to rotate back and forth over a second angular amplitude A2 in order to move (scan) the focused emission light spot over the imaging plane of the image sensor, while the focussed illumination light spot is moved (scanned) over the sample by the scanning mirror. The actuators and/or electro-motors of the scanning and rescanning mirror may be controlled by a computer system 120 so that the mirrors can be moved synchronously. Here, the frequency of the back and forth rotation of the scanning mirror is identical to the frequency of the back and forth rotation of the rescanning mirror.

In an embodiment, computer system may be configured as a central computer for centrally controlling the actuators and the optics. In another embodiment, the computer system may be a distributed system wherein different processor may control different parts of the system. For example, in an embodiment, each scanning mirror may be controlled by a separate processor in order to provide fast low delay control of the scanning mirrors.

Thus, when the scanning mirror is controlled by the computer system to scan an area of the sample by the moving illumination light spot, the rescanning mirror is controlled by the computer system to scan associated areas of pixels of the imaging plane of the image sensor so that the pixels are exposed by moving emission lights spots. When a sample is optically excited by the illumination beam that moves in accordance with a illumination pattern over the sample, emission light originating from the excited areas is imaged onto the imagining plane.

The structured light microscopy (SIM) module 132 may control the scanning and rescanning mirror in order to expose the sample multiple times, each time using a different illumination pattern, typically a periodic illumination pattern, so that during an exposure some parts of the samples are not exposed to the illumination light and other parts are. Examples of illumination patterns are described hereunder in more detail with reference to Fig. 2 and Fig. 3.

For each of these multiple exposures, the spatial position of said illumination pattern is spatially shifted and/or rotated with respect to a reference position associated with the sample. The exposure of the sample includes controlling a scanning mirror of a first optical system to move (scan) at least one focused illumination light signal through the sample in accordance with the illumination pattern, wherein the focused illumination light signal causes one or more optical excitations in the sample, the light originating from said optical excitations forming an emission light signal.

During the exposure of the sample, multiple images are captured, wherein each image is associated with one of the multiple exposures. Here, during the capturing of an image the rescanning mirror is controlled to reflect the emission light signal towards the imaging system and to move (scan) a focused emission light signal in accordance with the illumination pattern over the imaging plane of the imaging system. Thereafter, the SIM module may construct a high resolution image on the basis of the multiple captured images.

In an embodiment, the second angular amplitude A2 of the rescanning mirror 114 may be larger than the first angular amplitude Al of the scanning mirror 108 so that highresolution images of the scanned sample area can be obtained. For example, in an embodiment, the second angular amplitude may be twice the first angular amplitude, so that the reconstruction of the image can be improved when compared to the case when the second angular amplitude is chosen to be equal to the first angular amplitude.

The light source 100 may comprise one or more light sources, e.g. one or more lasers, and one or more light filters. In an embodiment, the light source may be connected to the computer system. The one or more light sources and filters may be used to control the wavelengths or bands of wavelengths the illumination light comprises. In an embodiment, the light source may be configured to generate white light. In another embodiment, the light source may be configured to generate light consists of two or more predetermined wavelengths or bands of wavelength. For example, in an embodiment, the light source may generate light of a first wavelength selected from the blue band of the visible spectrum and light of a second wavelength selected from the yellow band of the visible spectrum.

The optical elements of optical system 132 may be configured to control different wavelengths. For example, mirror 104 may reflect the illumination light onto dichroic mirror 106 that may be configured to reflect light of a first group of wavelengths, e.g. blue light and yellow light. Dichroic mirror 106 may be further configured to pass light of a second group of wavelengths, e.g. red and green light. This way, the illumination light may be reflected by the dichroic mirror onto the scanning mirror .

In an embodiment, the illumination light may comprise light of a first wavelength, e.g. yellow light, which may cause a first type of optical excitations in the sample, the first type of excitations generating first emission light. Further, the illumination light may comprise light of a second wavelength, e.g. blue light, which may cause a second type of optical excitations in the sample, the second type of excitations generating second emission light. Thus the abovementioned first group of wavelengths, which may be reflected by dichroic mirror 106, may comprise said first wavelength and said second wavelength.

In an embodiment, sample 122 may be a material, e.g. a biological material, that is imaged using a reflective microscopy technique. In an embodiment, sample 112 may be an optically active sample. For example, in an embodiment, the sample may be a material, e.g. a biological material, comprising one or more types of fluorescent/luminescent materials, such as fluorophores. In another embodiments, the optically active sample may be non-luminescent optical active material. For example, the sample may be a material that can be imaged on the basis of second harmonics generation (SHG) microscopy or third harmonics generation (THG) microscopy. In yet a further embodiment, the sample may be a material that can be imaged using a Rahman microscopy technique, such as Coherent Anti-Stokes Raman Scattering (CARS) microscopy.

For example, the sample comprise at least a first type of fluorophores, e.g. red fluorophores, and a second type of fluorophores, e.g. green fluorophores. The light of the first wavelength of the illumination light may cause the first fluorophores in the sample to emit the first emission light and the light of the second wavelength may cause the second fluorophores in the sample to emit second emission light.

Hence, the emission light may comprise light that is emitted as a result of photoluminescence, preferably fluorescence. In an embodiment, a blue light component of the illumination light may cause green fluorophores at the focused illumination light spot to emit green light. Similarly, yellow light component of the illumination light may cause red fluorophores at the focused illumination light spot to emit red light. In this example, the emission light thus comprises green light and red light.

Hence, from the above it follows that different photo-luminescent sites which are sensitive to light of different wavelengths and which - in response - emit light of different wavelengths, may be used. This way, the emission light originating from the sample may include optical information associated with different excitation sites of the sample. In an embodiment, filters such as color filters (not shown) may be positioned in front of the imaging system. A color filter may be configured as a bandpass filter that is configured to pass light in a predetermined band of wavelengths .

The scanning mirror both directs the illumination light signal from the light source 100 towards the sample 112 as well as the emission light signal from the sample 112 towards the rescanning mirror 114. This way, the movement of the emission light signal originating from the sample is neutralized by the movement of the scanning mirror. Due to the movement of the scanning mirror, the path of emission light from scanning mirror to rescanning mirror is static. In other words, the scanning mirror 108 descans the emission light 124 and reflects the emission light as a static emission light beam towards the rescanning mirror. Further, the path of the emission light between rescanning mirror and light detector may be dynamic during scanning due to the movement of the rescanning mirror .

Fig. 2A-2C depict a process of illuminating a sample with an illumination pattern using a scanning microscopy system according to an embodiment of the invention. In particular, Fig. 2A depicts a top view of a sample 206 wherein a position in the sample may be defined on the basis of a coordinate system, e.g. a 3D Cartesian coordinate system x,y,z. The origin of the coordinate system in the sample may be calibrated with respect to a reference position 202i so that after calibration the coordinate system may be used to accurately position the light spot in the sample, which may have lateral dimensions in the x,y plane and an axial dimension in the z-direction. The reference position may be a position on the sample or associated with the sample, e.g. a position on the sample (holder), e.g. an (optical) marker that is used by the system to position and align the sample within respect to the optical system, in particular the optical axis of the optical system, of the scanning microscope.

Fig. 2A further depicts part of an illumination pattern 204i, in this example a periodic arrangement of parallel lines, which is formed by writing the illumination pattern into the sample. To this end, the computer system, may generate or receive control information for controlling the scanning microscope, wherein the control information defines a plurality of different periodic patterns, e.g. a plurality of different stripe patterns, each being defined using one or more pattern parameters, e.g. the number of stripes, the length of the stripes, the spatial frequency, the direction of the periodic tripe pattern (periodicity direction), an initial phase of the pattern, an illumination light intensity, etc.

The computer system may use the control information to control the scanning mirror for writing the multiple illumination patterns in the sample. Each exposure to an illumination pattern causes one or more optical excitations in the sample and an emission light signal, formed on the basis of the light originating from said optical excitations, may be captured by the imaging system.

When looking at a cross-section 200 of the sample along a direction perpendicular to the scanning direction (in this case parallel to the x-direction), a periodic intensity function identifying high and low illuminated areas in the xdirection of the sample can be identified, wherein the spatial period of the function is identified by a parameter d.

Due to the fact that a focused illumination spot 208i is shaped according to a certain point spread function (PSF), the intensity pattern in a cross section 200 of the sample in the direction of the periodicity may be described as a sinelike function rather than a square wave function with sharp edges. Hence, the periodic intensity function of the first exposure in Fig. 2A may be expressed as a sinusoid function 2011 and an initial phase φο :

/(r) = l_Q

A ^{1 +} 2^S1

wherein I_o is the averaged intensity, A is measure of the modulation depth, 1/d the spatial frequency and φο an (initial) phase of the intensity function. The initial phase φο is directly related to the position of the illumination structure in the sample as is illustrated in Fig. 2A.

The point spread function may cause a certain blurring so that when the spatial period is decreased, the modulation depth of the periodic intensity function will become smaller. The decrease of the modulation depth with increasing spatial frequency is a general problem with conventional SIM that typically uses a grating in order to expose a sample with a periodic illumination structure. Due to the decrease modulation depth, it is no longer possible to determine pattern parameters such as the initial phase and the frequency on the basis of the captured images.

This problem is solved by the methods and systems described in this application. The scanning microscope according to the invention may accurately control the position of a focused illumination light spot in the sample, a periodic 2D or 3D illumination pattern can be accurately written into the sample using the control information.

Pattern parameters, including the initial phase and the spatial frequency, defining the different periodic patterns - which are determined before exposing the sample may be used in the reconstruction algorithm. The invention thus eliminates or at least substantially reduces the need for a posteriori image analysis in order to determine illumination light pattern parameters that are used by the reconstruction algorithm. This way, light patterns may be used that have a higher spatial frequency than the light patterns that are used in SIM systems known in the prior art.

Once a first illumination pattern is written into the sample and a corresponding image of the illuminated sample is captured by the imaging system and stored in a memory storage, the computer system may repeat the process of exposure and image capturing several times wherein each time, the illumination pattern is shifted and/or rotated with respect to the reference position. As shown in Fig. 2B and 2C, shifted versions of the periodic stripe structure may be written into the sample. The shifted illumination patterns of Fig. 2B and 2C may be expressed by similar sine functions 2012,3 having initial phases φι and tpc respectively. Hence, the spatial shift may be expressed in terms of a spatial phase shift of the periodic intensity function in the x-direction.

Fig. 3 depicts different illumination patterns for use in an image reconstruction algorithm. As shown in Fig. 3, first the sample may be exposed to a first set of light patterns 3061-3, e.g. a set of shifted periodic tripe patterns wherein the stripes may have a predetermined orientation in the 2D plane, e.g. aligned to the y-axis as e.g. described with reference to Fig. 2A-2C. Thereafter, the computer system may repeat the exposure of the sample using a second set of light patterns 3041-3 wherein the second set is a rotated version (in this example 45 degrees) of the first set. Similar, the process may be repeated using a third set 3061-3 which is a rotated version of the first or second set). Each pattern in the sets of patterns is defined by pattern parameters such as an initial phase which may be directly used in the reconstruction algorithm.

Although the examples in this disclosure are described with reference to a periodic stripe pattern, many other illumination patterns may be used. In an embodiment, the illumination pattern may have a periodic 2D pattern that has a periodicity in one direction (e.g. such as the pattern depicted in Fig. 2A). In another embodiment, the illumination pattern may be a periodic 2D pattern that has a periodicity in two directions, e.g. an array of blocks having a periodic in the x and y direction. In yet a further embodiment, the illumination pattern may be a 3D periodic pattern. The sample may be exposed at different focus depths in the (axial) zdirection. Hence, in that case, the illuminated areas are not only separated in the x,y plane shown, but also separated in the axial direction.

Fig. 4 depicts a process of reconstructing an image on the basis of images obtained by structured illumination microscopy according to an embodiment of the invention. In an embodiment, the reconstruction process may be executed by computer system of the scanning microscopy system as for example descripted with reference to Fig. 1.

As described above, the illumination pattern may be formed by a periodic illumination pattern which may be represented by a sinusoid, which in Fourier space may be represented by three delta functions 402 as shown in Fig. 4. The Fourier transform may be expressed as follows:

I(k) = 5(/c) + y e^i(p°5 (k including a first delta function at frequency k=-l/d, a second delta function at frequency k=0 with an intensity Io and a third delta function at frequency k=+l/d.

The optical response of the sample to the structured illumination is a product of the applied light intensity pattern, I(r), and the fluorophore distribution, S(r). Hence, the optical response in reciprocal space is a convolution of the Fourier transform of the illumination pattern 402 and the Fourier transform of the (unknown) fluorophore structure S(k) 404, which represents an image that needs to be reconstructed. The optical response 406 can be represented by:

OR(k) = I(k) ®S(k) =

S(k) - S

S

The microscope, which may comprise one or more lenses, that may be used to capture image data of the optical response may be characterized by an Optical Transfer Function OTF(k) 408. The OTF shows that the microscope system effectively acts as a low pass filter, attenuating high frequencies. The dashed vertical lines indicate a cut-off frequency. Frequencies higher than this cut-off frequency will not pass through the example microscope system.

The Fourier transform of the captured images D(k) 410 is a product (in reciprocal space) of the optical response OR(k) 406 and the OTF(k) 408. This relation may be expressed by the following formula:

D(k) = OR(k) * OTF(k) =

* OTF(k) wherein the system does not capture image data associated with frequencies higher than the cut-off frequency of the microscope system.

The Fourier transform of the illumination pattern is known. Further, the frequency 1/d and initial phase cpo are pattern parameters which are determined prior to the exposure and can be used directly in the computation.

This way, it is mathematically possible to compute the unknown Fourier transform of the fluorophore structure S(k) for a range of frequencies including frequencies exceeding the cut-off frequencies of the OTF of the microscope system. This process may be executed on the basis of a known image reconstruction algorithm 412. The computed Fourier transform 414 of the fluorophore structure comprises frequencies that are higher than the cut-off frequency of the OTF of the microscope. This computed Fourier transform may be used to reconstruct a high resolution image.

In the above described example, the reconstructed image comprises enhanced resolution in only one direction, namely the direction in which the periodicity exists. In order to obtain enhance resolution in other directions as well, the above described method has to be repeated using illumination patterns having different periodicity directions, i.e. patterns that are periodic in different directions (as e.g. shown in Fig. 3).

Hence, in the image reconstruction process described with reference to Fig. 4, pattern parameters such as the initial phase and the spatial frequencies defining the patterns that are used for exposing the sample are used in the reconstruction process. As the scanning microscope is capable of accurately controlling the position of the illumination light spot in the sample, a high resolution image can be reconstructed .

Fig. 5A-5C depicts various illumination light spots for use in a scanning microscopy system according an embodiment of the invention. In particular, FIG. 5A-5C illustrates differently shaped illumination light spots, i.e. illumination light spots with different light intensity profiles, that can be used to illuminate areas on the sample,

i.e. write the periodic illumination pattern onto or into the sample. In this figure, the darker areas are associated with high light intensities and the lighter areas with low light intensities. The illumination light spots move in the direction of the arrows, i.e. in the writing direction. The dashed lines indicate one or more areas of the plurality of areas, i.e. the illuminated areas on the sample, after the illumination light spot has moved over the sample in the direction as specified by the arrow.

FIG. 5A shows an illumination light spot that has a simple Gaussian shaped light intensity profile. FIG. 5B shows an illumination light spot that comprises a predetermined light intensity pattern. In this embodiment, the illumination source of the scanning microscopy system and the optics associated with the illumination source may be adapted to generate an illumination light spot 5042 comprising at least two intensity maxima wherein d defines the distance between the maxima. When using such illumination light spot, at least two areas on the sample may be illuminated simultaneously thus proving an efficient way of writing multiple parts of the periodic illumination structure simultaneously.

Alternatively and/or additionally, in an embodiment, the illumination light spot may be shaped, e.g. oval instead of circular so that the width of the spot in the scanning direction is larger than the width of the spot in the direction perpendicular to the scanning direction. Minimizing the width of the illumination spot perpendicular to the scanning direction (writing direction) allows writing of periodic illumination structures that have higher spatial frequency which is beneficial for achieving higher resolutions .

In case the illumination light spot is shaped into a predetermined shape, e.g. an oval shape as shown in Fig. 5B, the orientation of the shaped light spot has to be changed when changing the scanning direction so that the width of the spot perpendicular to the scanning direction is always the smallest width. This way, a fine resolution of the illumination pattern is ensured in every scanning direction as shown in Fig. 5B.

Fig. 5C depicts a further embodiment wherein the illumination light spot comprises a plurality of maxima so that a plurality of lines are generated when moving the illumination light sport in predetermined directions. For example, the maxima may be arranged in a square configuration, so that when moving the light sport in a first direction a plurality of lines are simulations written onto the sample and when moving the light sport in a second direction a plurality of lines are simultaneously written onto the sample.

In the example of Fig. 5C, the plurality of maxima may be arranged in a configuration having three axis of rotation symmetry, i.e. the x-axis, y-axis and an 45 degree axis. When the scanning direction of the illumination light spot is parallel to one of the directions of axis of symmetry, the same striped pattern can be generated.

The shape of the illumination light spot and the number of maxima in the illumination light spot may be controlled using an element that can shape the phase-profile and/or intensity profile of the wave front of the excitation laser beam, such as a deformable mirror of a spatial light modulator. Such optical element is controlled by a computer with some control software.

Fig. 6 depicts a scanning microscopy system according to another embodiment of the invention. In particular, Fig. 6 depicts a scanning microscopy system similar to the system described with reference to Fig. 1, comprising a light source, a sample 612, an imaging system 618 and optical elements including a mirror 604, dichroic mirror 606, scanning mirror 608, focusing lens 610, rescanning mirror 614 and focusing lens 616.

In contrast with the system of Fig. 1, the system in Fig. 6 comprises an optical arrangement 621 for enabling confocal measurements wherein the optical arrangement may comprise a pinhole 623. The scanning microscopy may be configured to focus illumination light spot 613 in a plane of interest P. Additionally, the system may be configured to arranged to focus emission light from the plane of interest P onto a plane 623 P'. For example, the system may comprise one or more optical elements, e.g. lenses 620 and 622, arranged to focus emission light from the plane of interest P onto plane P'. Plane P' thus may be a confocal conjugate plane of plane of interest P.

In a further embodiment, the scanning microscopy system may comprise a point spread function (PSF) module 603 that is configured to modify the illumination light spot by modifying the shape of the illumination light spot and/or introducing a predetermined arrangement of light intensity maxima in the illumination light spot as e.g. described with reference to Fig. 5A-5C.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises and/or comprising, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CONCLUSIONS

A method of forming a high-resolution image of a specimen using a processor-controlled scan microscope, comprising:

the processor receiving or generating control information for controlling a scanning microscope, wherein the control information defines a plurality of different periodic patterns based on one or more pattern parameters, the one or more pattern parameters preferably comprising at least one of: a spatial frequency, a periodicity direction, an initial phase;

using the control information by the processor to control the scanning microscope to expose the specimen to multiple exposure patterns, each exposure to an exposure pattern causing one or more optical excitations in the specimen, the light from the optical excitations causing a forms an emission light signal;

controlling by the processor of an imaging system to capture multiple images, each image associated with an emission light signal from one of the multiple exposures, and, using the processor of an image reconstruction algorithm to form a high image image resolution based on at least one of the one or more pattern parameters and the captured images.

The method of claim 1, wherein the exposure of the specimen to a plurality of illumination patterns comprises:

moving the position of a focused exposure light spot of the scanning microscope in accordance with each of the plurality of periodic patterns.

The method of claim 2, wherein a reference position associated with the specimen defines a coordinate system, the coordinate system providing a position of the exposure light spot relative to the reference position.

The method according to claims 2 or 3, wherein the focused illumination light spot comprises a predetermined light intensity pattern comprising at least two spatially, preferably laterally or axially arranged, light intensity maxima, wherein the spatially arranged light intensity maxima define a spatial frequency of the illumination pattern.

The method of claim 4, wherein the predetermined light intensity pattern comprises at least a 2D arrangement of two or more light intensity maxima; and / or wherein the predetermined light intensity pattern comprises a 3D arrangement of a plurality of light intensity maxima.

The method according to claims 2 or 3, wherein the focused exposure light spot is a shaped exposure light spot, the light spot formed comprising at least a first width and at least a second width smaller than the first width, the second width preferably during scanning is arranged perpendicular to the direction in which the focused exposure light spot moves.

The method according to one or more of claims

1-6, wherein capturing multiple images comprises:

controlling a rescan mirror of the scanning microscope to reflect the emission light signal to an imaging system and to move a focused emission light signal in accordance with the exposure pattern over the imaging surface of the imaging system.

The method of claim 7, wherein controlling the scan and rescan mirror comprises rotating the scan mirror and the rescan mirror back and forth by a predetermined first angle amplitude and a second angle amplitude, respectively; wherein preferably the ratio between the first and second angular amplitude is unequal to one; more preferably the second angle amplitude being selected as twice the first angle amplitude.

The method of any one of claims 1-8, wherein forming a high-resolution image comprises:

determining a Fourier transformation of the various multiple exposure patterns based on the one or more pattern parameters.

The method as claimed in one or more of claims 1-9, wherein the scanning microscope comprises at least one pinhole for filtering out of focus light from the exposure light signal, wherein preferably the scanning microscope comprises one or more optical elements arranged around an illumination light spot projecting a plane of interest of the specimen onto a confocal conjugated plane of the plane of interest.

The method of any one of claims

1-9, wherein the control information is arranged to control one or more actuators for controlling a scan mirror and / or rescan mirror to move a focused exposure light spot in a lateral direction in the specimen; and / or, to control one or more optical elements, preferably one or more lenses, to move the focused exposure light spot in an axial direction in the sample.

12. A scanning microscopy system for forming a high-resolution image comprising:

a computer adapted to control the scanning microscope, the computer comprising a computer readable medium with at least a part of a program embodied therein; and, a computer-readable storage medium containing computer-readable program code, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to executing the computer-readable program code, the processor is arranged to perform executable operations, comprising:

receiving or generating control information for controlling a scanning microscope, wherein the control information defines a plurality of different periodic patterns based on one or more pattern parameters, the one or more pattern parameters preferably comprising at least one of: a spatial frequency, a periodicity direction, an initial phase;

using the control information to control the scanning microscope to expose the specimen to multiple exposure patterns, each exposure to an exposure pattern causing one or more optical excitations in the specimen, the light from the optical excitations forming an emission light signal;

controlling an image system to capture multiple images, each image associated with an emission light signal from one of the multiple exposures, and using an image reconstruction algorithm to form a high resolution image based on at least one one of the one or more pattern parameters and the captured images.

A scanning microscope according to claim 12, wherein the executable operations further comprise:

controlling a rescan mirror of the scanning microscope to reflect the emission light signal to an imaging system and to move a focused emission light signal in accordance with the exposure pattern over the imaging surface of the imaging system, preferably controlling the scanning and rescan mirror to rotate the scan mirror and the rescan mirror back and forth by a predetermined first angle amplitude and a second angle amplitude, respectively; more preferably, the ratio between the first and second angular amplitude is unequal to one; wherein even more preferably the second angle amplitude is selected as twice the first angle amplitude.

A control module for controlling a scan microscopy system comprising:

a computer adapted to control the scanning microscope, the computer comprising a computer readable medium with at least a part of a program embodied therein; and, a computer-readable storage medium having computer-readable program code embodied therein, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to executing the computer-readable program code, the processor is arranged to perform executable operations, comprising:

15. Computer program product comprising parts of software

5 code, when executed in the memory of a computer, to perform the method steps according to one or more of claims 1-11.

4/6

Reconstructed Fourier transform kJ

Ώ ώ

CL ώ aseees sa