Resolution enhancement of digital holographic microscopy via synthetic aperture: a review

Optical microscopy is a powerful tool for detecting subtle structures of samples, playing an irreplaceable role in many fields. However, traditional optical microscopy can only obtain the amplitude information of samples, and transparent samples, such as living cells cannot be investigated. Although fluorescence microscopy can selectively render and highlight the structures of interest by prior tagging with fluorescent markers, the phototoxicity and photobleaching of fluorescence tagging make it difficult to continuously observe living cells over a long time. Thus, there is an urgent need for label-free microscopic tools able to follow organelles in live cells. Digital holographic microscopy (DHM) ^1-6, which combines digital holography and microscopy, is a label-free, quantitative phase microscopy approach. A typical setup for DHM is shown in Fig. 1a, in which a magnified object wave and a reference wave interfere with each other, and the generated off-axis hologram or phase-shifting holograms are recorded by a CCD camera. From the off-axis hologram (Fig. 1b), both the amplitude and phase images (Fig. 1c, d) of a sample can be obtained simultaneously. As an alternative, inline lensless DHM can help recover the phase information with an iterative algorithm without using an independent reference wave^{7, 8}. Refocusing of the reconstructed images can further be performed digitally using the obtained complex amplitude of the object wave^{9, 10}. In general, DHM can not only observe transparent samples with high endogenous contrast, but also quantitatively assess the thickness or refractive index distributions of these samples. Therefore, DHM has been widely applied in industrial inspection^{11, 12}, visualization of liquid/gas flows¹³, biomedical imaging¹⁴, etc.

Fig. 1  Principle of digital holographic microscopy (DHM). a Schematic setup of off-axis (left) and in-line DHM (right). b Exemplary hologram of off-axis DHM. c, d Amplitude and phase images of HeLa cells reconstructed from b. Scale bar in c is adapted with permission from Ref. 76 ©The Optical Society.

The spatial resolution is of great importance for DHM because it determines the smallest structures that can be resolved. As illustrated in Fig. 2a, when two infinitely small points separated by a Rayleigh distance of 0.61λ/NA are imaged by a diffraction-limited imaging system, two Airy patterns are generated in the imaging plane, whose complex amplitudes are denoted as O₁ and O₂, respectively. Here, λ and NA indicate the illumination wavelength and the sum numerical aperture (NA) of the imaging system (NA_imag) and illumination system (NA_illum), respectively. For incoherent illumination, the intensity of the image is linearly related to the intensity emitted from the sample, that is, |O₁|²+|O₂|². The two patterns are considered to be resolved if the center of the intensity pattern of O₁ coincides with the first zero of the pattern of O₂ (Fig. 2b). For coherent imaging (e.g., DHM), a linear relationship holds between the input complex amplitude and the output complex amplitude. In this case, the intensity of the image corresponds to |O₁+O₂|², which is dependent on the phase difference Δ$\varphi $ between O₁ and O₂. It is evident from Fig. 2c that for Δ$\varphi $ = 0 (in-phase), the Airy patterns of O₁ and O₂ are considered unresolved. However, for Δ$\varphi $ = π (anti-phase), the Airy patterns of O₁ and O₂ are well separated. To conclude, there is no easy criterion for evaluating the resolution of coherent microscopy (DHM), considering that the resolution depends on both the microscope and the phase of the sample. Nevertheless, we recommend the following:

Fig. 2  Resolution criteria in incoherent and coherent systems. a Intensity distribution of the Airy patterns for the complex amplitude distributions, O₁ and O₂, obtained by imaging two infinitely small point sources through a diffraction-limited system; the superimposed intensities of O₁ and O₂ in incoherent microscopy b and coherent microscopy c. In-phase and anti-phase in c indicates that the cases O₁ and O₂ have a phase difference of 0 and π, respectively.

for evaluating the resolving power of optical microscopy. Factor κ₁ is determined by the experimental parameters, such as the coherent noise level and the SNR of the detector, etc^{3, 15}. Herein, we assign κ₁ = 0.82^{16, 17}, for which the principal intensity maximum of pattern O₁ coincided with the first minimum of the in-phase O₂. This suggests two options for enhancing the spatial resolution of the DHM. The first is the use of a shorter wavelength, for example, 193-nm¹⁸ light was used to enhance the spatial resolution of the DHM. The other is to enlarge the NAs of both the illumination and the recording systems.

In addition to the resolution, the field of view (FOV) is also a vital parameter of an imaging system. The ratio between these two parameters determines the space-bandwidth product (SBP) of the imaging system^{19, 20}. Often, a DHM system has a limited SBP, and consequently, there is an inherent trade-off between resolution and FOV. This means that a reconstructed image with a higher resolution has a smaller FOV. Just as an example, a standard 20 × /0.4 microscope lens with a resolution limit of 0.8 μm and a circular FOV of 1.1 mm in diameter, provides a SBP of ~7 megapixels²¹. However, microscopists always expect to have a higher SBP in the sense of an increasing resolution while maintaining FOV. There are many degrees of freedom, including spatial/temporal resolution, FOV, polarization, spectrum, etc., in an imaging system. We can convert or trade some unused degrees of freedom (e.g., time) for an increase in SBP. In the past few decades, computational microscopy methods have emerged, generating images with high resolution and a wide FOV by using modulated illumination or applying other physical manipulations during the imaging process. By synthesizing apertures using the object information obtained under different modulation/manipulation operations, the SPB limitation of conventional microscopes (including DHM) can be bypassed^{22, 23}.

This paper aims to review the resolution enhancement approaches of DHM, which are classified into three types: 1. Enlarging the NA of illumination via oblique illumination, structured illumination, and speckle illumination; 2. Enlarging the NA of the recording system via hologram extrapolation, hologram expansion, and pixel super-resolution. 3. Artificial intelligence (AI) assistance resolution enhancement approaches for DHM. This paper will cover the basic principles, implementation schemes, and exemplary results of the different approaches. We conclude by providing a summary and future outlook of resolution-enhancement approaches for DHM.

A microscopic imaging system is commonly introduced to DHM, which provides a magnified and high-resolution image of the sample. A plane wave is often used for illumination, and only the spatial frequencies diffracted by the sample up to ~ NA_imag/λ can be transmitted through the limited aperture of the microscope lens (Fig. 3a, upper part), resulting in an upper limit for the spatial resolution in terms of 0.82λ/NA_imag. Therefore, using oblique illumination, structured illumination, or speckle illumination to provide a NA_illum is an effective method for further improving the resolution.

Fig. 3  DHM with oblique illumination. a Schematic of synthetic aperture via oblique illumination in different orientations. b schematics of setup of off-axis DHM with oblique illumination. Fig. 3b is adopted from Ref. 49.

Since the first implementations of oblique beam illumination in DHM in the 1970s^24-27, many approaches have been extensively reported on for replicating a larger aperture^28-48. When a sample is illuminated by an oblique wave, the high spatial frequency of the object wave in the opposite direction of the oblique illumination will be downshifted and, thus, will pass through the limited aperture of the imaging system (Fig. 3a, lower part). The downshifted frequencies are back-assembled to their original positions in the spectrum of the object, thus, synthesizing a wider spectrum than that of the NA-defined aperture³¹. The final resolution-enhanced image is obtained by an inverse Fourier transform (FT) of the synthesized spectrum. In comparison with conventional lens-based DHM with a resolution limit of 0.82λ/NA, DHM with oblique beam illumination has an enhanced spatial resolution, as described in Eq. 1.

Kuznetsova et al. and Schwarz et al. used oblique beam illumination to downshift the high-frequency components of the spectrum of the object, and introduced a conjugate carrier phase to upshift the spectrum content back to its proper region^{28, 50}. This process was repeated sequentially for both the x- and y-illumination directions, thus allowing for the transmission of extended frequency bandwidths of the 2D object, yielding a resolution gain with a factor of three. Mico et al. utilized a vertical-cavity surface-emitting laser (VCSEL) array to generate multi-directional oblique illumination of DHM and retained a resolution gain with a factor of five⁴⁷. The use of scanning elements has also been reported as a technological improvement for providing more flexible oblique beam illumination^{28, 33, 37, 51}, which has also been validated^{29, 31, 52} in comparison with the tilting of the illumination direction. For instance, Cheng et al. used a 2D Galvo-scanner to generate oblique illumination for DHM (Fig. 3b), achieving a two-fold isotropic resolution enhancement. ⁴⁹ Oblique beam illumination for resolution enhancement has also been implemented using a spatial light modulator (SLM) ^53-55 or a fiber bunch^56-60. Meanwhile, single-shot synthetic aperture DHM was realized by using an SLM to generate multiple non-coplanar illumination beams, and using coherence gating to avoid self-interference between these non-coplanar beams⁵³. Moreover, wavelength multiplexing techniques^34-36 were utilized to perform multiple oblique illuminations during a single exposure. Oblique beam illumination techniques have recently been implemented in a commercial upright Olympus microscope³⁹. Notably, the maximum synthetic numerical aperture is limited by NA<1 in imaging systems with dry objectives, and further resolution improvements to λ/3.7 were demonstrated using evanescent waves²⁹. In addition, axial rather than transversal resolution improvement has also been validated by SA generation⁶¹, and an application of the technique to edge processing was also reported⁶². It is interesting to mention that oblique illumination has also been applied to differential interference contrast (DIC) microscopy⁶³ and Zernike phase contrast microscopy⁶⁴.

Recently, oblique illumination was also applied to obtain 3D refractive index (RI) tomographic images of transparent or translucent samples via optical diffraction tomography (ODT)^{49, 65}. ODT enables the probing of 3-D RI maps of a sample by recording the complex wavefront diffracted while rotating the sample^{66, 67} or varying illumination angles^{68, 69}. Kim⁶⁸ adapted DHM to ODT by using a 2D galvanometric mirror to generate different illumination directions and consequently, reconstructed a 3D RI distribution of the samples with high lateral and axial resolutions. Ozcan⁷⁰ presented an on-chip ODT scheme that enables the imaging of a large volume of approximately 15 mm³, with a spatial resolution of < 1 μm ×1 μm × 3 μm. In 2020, Wang et al.⁷¹ reported a mechanical-scanning-free ODT configuration using self-accelerating Airy beams that are tilted along the beam path direction, providing a 3D volumetric view of the cells. Recently, Kus et al.⁷² proposed a real-time ODT system for recording a complete set of projections in a single hologram in a multiplexing manner by using a microlens, with which the tomographic images of living cells and flow cytometry could be observed clearly. In addition to advances in ODT hardware, artificial intelligence (AI)-based reconstruction algorithms have been reported to reduce the recording time and improve the quality of the reconstructed images⁷³.

Structured illumination microscopy (SIM) is a super-resolution optical microscopy technique that has the merits of being fast, minimally invasive, and has no requirement for fluorescent labeling^{74, 75}. Compared with conventional wide-field (WF) microscopy, SIM provides two-fold spatial resolution enhancement by illuminating the sample with a periodic pattern and recording the generated moiré pattern. Because of the moiré effect, the high spatial frequency components of the sample, which are not accessible in conventional microscopy, are shifted into the detectable domain and thus can be observed. However, the application of SIM has been limited to fluorescent samples. In 2010, Mudassar and Hussain⁶⁰ applied it to non-fluorescent scattering imaging, for which fringe patterns were generated by the interference between the beams carried by two fibers.

A DHM with structured illumination generated by an electric modulation device has been proposed. The fringe patterns were generated by a spatial light modulator (SLM)⁷⁶ in 2013 and by a digital micromirror device (DMD)⁷⁷ in 2015, whereby fringes with different orientations and phase shifts were projected without mechanical motion. In the implementation, four binary phase gratings rotated by m×45° (Fig. 4a) and generated by an SLM or a DMD are projected onto the sample, so that the sample is illuminated sequentially by four sinusoidal fringe patterns. After passing through a telescope system comprised of the microscope objective MO and lens L, the object wave interferes with a tilted reference wave R, and the generated holograms are recorded by a CCD camera, as shown in Fig. 4b. We denote with Ψ_mn the object wave generated by illuminating the sample with the grating as having the m-th orientation and n-th phase shifting. Thus, the recorded hologram can be described as I_mn = |R+ Ψ_mn|². From this intensity, the wave Ψ_mn can be reconstructed using standard reconstruction methods, as in off-axis DHM. Generally, Ψ_mn can be decomposed into three components, A_{m, −1}, A_{m, 0}, and A_m,1 corresponding to the −1^st, 0th, and +1^st diffraction orders of the illumination wave. When we assume that the phase shift increment for each shifting of the grating is α, Ψ_mn can be written as:

Fig. 4  DHM with structured illumination. a Schematic of structured illumination with different orientations and phase shifts. b Schematic setup of DHM with structured illumination. c Experimental results for resolution enhancement. Reconstructed phase images using plane wave illumination (top) and structured illumination (bottom). b is adopted from Ref. 77. and c is adopted from Ref. 76.

where γ₋₁, γ₀, and γ₁ denote the magnitudes of the diffraction orders. From Eq. 2, we can calculate the reconstructed object waves A_{m, −1}, A_m,0, and A_m,1 from the different diffraction orders:

Then, A_m,-1 and A_m,1 under each illumination direction are compensated for by the carrier phase due to the tilted propagation direction along the ±1^st diffraction orders, and synthesized together with A_m,0 in the Fourier plane to yield the synthetic spectrum. Finally, a focused image with enhanced resolution was retrieved by the inverse FT of the synthetic spectrum. Assuming that the angular aperture of the imaging system is NA_imag, the synthetic NA of the SI DHM is NA = NA_imag+sinθ_illum. Here, θ_illum denotes the illumination angle of the +1 or −1 diffraction orders, and the highest value of the sinθ_illum is, in turn, determined by the NA of the condenser lens. Fig. 4c validates the resolution enhancement of the structured illumination in comparison with on-axis illumination. Of note, the phase image at the bottom (obtained with structured illumination) has a better resolution compared with that on the top, and the two particles separated by 1.4 μm became distinguishable.

Recently, structured illumination has been widely applied to different variations of DHM for the purpose of imaging transparent samples with resolution enhancement^76-89. SI-based DHM was extended for resolution enhancement in the axial direction, aside from the lateral directions⁹⁰. An iterative reconstruction algorithm, as well as a PCA-based algorithm, were employed to obtain resolution-enhanced images using structured illumination with unknown phase shifts or free of phase shifts^{82-84, 89}. In 2021, an end-to-end deep-learning-based method, DL-SI-DHM, was proposed for improving the reconstruction efficiency and accuracy of SI-DHM⁹¹. In addition to synthetic-aperture phase imaging, Shin et al. in 2015 and Chowdhury et al. in 2017 utilized DMD- and SLM-based structured illumination to obtain 3D refractive index tomographic images of samples^{77, 92}.

Speckle illumination has been known since the invention of the laser in 1960⁹³. Speckle illumination often reduces the signal-to-noise ratio (SNR) in coherent imaging. Speckle fields can be understood as a combination of plane waves with various random illumination directions. These oblique plane waves shift the spectrum of the object in the reciprocal plane, allowing for access to additional spatial frequencies. Thus, speckles can be used to enhance the spatial resolution and reduce the coherent noise in DHM or other coherent imaging^{37, 94-97}.

Fig. 5 shows the optical setup of DHM with a speckle illumination according to Ref. 95. The speckle field was generated by an SLM and projected using a telescopic system L₁-MO₁ in the sample plane, where $ {A}_{Speckle}^{i} $ is the ith speckle illumination and $ {A}_{o}^{i} $ refers to the complex amplitudes of the object wave when the sample was illuminated with $ {A}_{Speckle}^{i} $. When $ {A}_{o}^{i} $ interferes with a reference wave R, the hologram can be denoted as $ {I}_{o}^{i}={{|A}_{o}^{i}+R|}^{2} $. The complex amplitude of the speckle field $ {A}_{Speckle}^{i} $ can be reconstructed from the hologram recorded in the absence of a sample. Often, the holographic resolution-enhanced image can be obtained by averaging multiple angle-dependent E-field images O_ave = $ 1/\mathrm{N}· \sum _{i=1}^{N}{{A}_{o}^{i}/A}_{Speckle}^{i} $⁹⁶. It is worth mentioning that the resolution enhancement capability of such an averaging operation will be lower when the speckle field has an unequal weight or power over the entire frequency spectrum.

Fig. 5  Schematic of DHM with speckle illumination.

To overcome this limitation, an iterative reconstruction method⁹⁵ was proposed for synthesizing a larger NA. The key idea of the iteration is to propagate the object wave between the image plane and the CCD plane, updating the speckle illumination in the image plane and the object wave intensity (with the recorded one) in the CCD plane. The iterative method enhances not only the resolution but also the SNR when the reconstructed object wavefronts from different speckle-based holograms are averaged. Utilizing SLM for speckle field illumination avoids mechanical movements and allows for a high repeatability in comparison with dynamic devices. Time-changing speckle illumination is similar to structured illumination. The concept in common is the encoding of an object with an a priori known high-resolution pattern that folds the high spatial frequencies into the low bandwidth range. The decoding patterns should be matched to the time-changing encoding patterns.

Similar to speckle illumination, a scattering medium can also be placed before the sample to enhance the spatial resolution and FOV^{98, 99}. Multiple scattering of the medium is a deterministic process that is described by a transmission matrix (TM) in transmission geometry. TM-based imaging has been introduced in DHM, yielding an improved resolution beyond what an objective-lens-based imaging system can achieve⁹⁹. Similarly, a single multimode optical fiber was demonstrated for scanner-free and wide-field endoscopic imaging⁹⁶. Beyond resolution enhancement, Baek et al. demonstrated in 2018 that a high-resolution, long working distance, reference-free DHM can be performed by using a scattering layer to replace the conventional imaging lens¹⁰⁰. By exploiting the randomness of a multiple-scattering layer, it allows for holographic imaging of a microscopic object without introducing a reference beam. The authors experimentally demonstrated a high NA (0.7) with a working distance of 13 mm.

Compared to DHM with an imaging system, DHM without using an imaging lens (lens-free DHM) possesses three significant advantages. The first aspect is its compactness and low-cost because it does not employ an objective. The second is a large FOV. The third is the high compatibility and ease of integration with microfluidic and on-chip devices. In the following sections, we review the resolution enhancement of lens-free DHM via hologram extrapolation, hologram synthesis, and pixel super-resolution techniques.

As a major type of lens-free DHM, lensless DHM can be traced back to an inline architecture proposed by Gabor⁸. In Gabor, in lensless DHM^{8, 101, 102} (Fig. 6a) a sample is illuminated by a spherical wave. The wave diffracted by the sample interferes with the non-diffracted wave, which also acts as a reference wave. Direct reconstruction of the generated hologram suffers from overlapping of the DC term and the twin image. Iterative approaches^{7, 103, 104}, and compressive sensing¹⁰⁵ have been invented to retrieve phase information from an oversampled diffraction pattern. The achievable spatial resolution is believed to be limited by the size of the hologram^{8, 104}:

Fig. 6  Resolution enhancement of lensless DHM by self-extrapolation of holograms. a Experimental setup and results. a₁, hologram with zero-padding. a₂, hologram after 100 iterations. a₃ and a₄ refer to the reconstructed image without and with self-extrapolation, respectively. b Illustration of iterative reconstruction of a hologram by self-extrapolation. The image is adopted from Ref. 104.

where λ is the wavelength, z is the distance between the object and the screen, N is the number of pixels, and Δ_S is the pixel size of the detector. In 2013, resolution enhancement of lensless DHM was reported by extrapolation of the hologram^104,, and later, this method was applied to terahertz inline lensless holography^{106, 107}. The flowchart of the resolution enhancement routine is shown in Fig. 6b, and includes the following steps:

(1) Initialize the object wave U(x_S, y_S) in the hologram plane by taking the square root of the hologram intensity H₀ of N₀×N₀ and padding it with zeros to a matrix of N × N with N > N₀ (see Fig. 6a₁). In contrast, random noise can be added to the padded area.

(2) Propagate the padded wave U(x_S, y_S) to the object plane by using the propagation integral⁷, yielding o(x_o, y_o).

(3) In the object plane, a constraint concerning the finite size and positive absorption of the sample is applied to o(x_o, y_o), resulting in $ {o}^{\text{'}} $(x_o, y_o).

(4) Propagate $ {o}^{\text{'}} $(x_o, y_o) in the hologram plane and replace the amplitude of $ {o}^{\text{'}} $(x_o, y_o) with the square root of the hologram intensity H0 within the central N₀×N₀ area (see Fig. 6a₂).

Once the above procedures have been repeated over 500 iterations, the complex amplitude of the object wave can be retrieved, which has a resolution of σ = λz/(NΔ_S) instead of λz/(N₀Δ_S). It is clear that the image reconstructed with hologram extrapolation (Fig. 6a₄) has finer structures than that obtained without using self-extrapolation (Fig. 6a₃). Quantitatively, the resolution was enhanced from 5.9 μm to 1.8 μm by padding a hologram with 300 × 300 to 1000 × 1000 pixels¹⁰⁸. The reason for the resolution enhancement by hologram extrapolation is that the hologram itself contains a series of wavelets constituting the wavefront, which can be fitted better through extrapolation and yield a larger effective hologram¹⁰⁴. This hypothesis is confirmed by a closer look at the intensity distributions ((iv) in Fig. 6a₂) in the hologram plane.

In addition to inline lensless DHM, DHM, which records the interference between an object wave and an independent reference wave can reconstruct the phase of a sample free of twin-image disturbances. Fig. 7a shows a standard lens-free DHM configuration using an independent reference wave^{109, 110}. Fig. 7b shows a modified lensless DHM configuration by the insertion of an SLM into the classical Gabor architecture¹⁰². The SLM is used to perform phase shifting of the DC term of the object wave. The complex amplitude distribution of the object wave passing through a sample can be recovered using a standard phase-shifting reconstruction algorithm. For the two types of lens-free DHM configurations (with and without an independent reference wave), the spatial resolution is limited by the size of the hologram, as shown in Eq. 4. In 2002, resolution enhancement of lens-free DHM was first demonstrated¹¹¹ by generating an expanded hologram with multiple sub-holograms. Each sub-hologram covers a different frequency spectrum of the input object. When the reconstructed image of each hologram is synthesized, an image with an enlarged FOV and an enhanced spatial resolution can be obtained^{109, 110}. Generally, hologram synthesis of lens-free DHM can be performed shifting the digital camera¹⁰⁹ or sample^{110, 112}, using oblique beam illuminations¹¹³, or inserting a diffraction grating between the object and digital camera^114,115. The above-mentioned three strategies are equivalent and generate the same output, which is a larger synthetic hologram that contains a larger sample spectrum.

Fig. 7  Resolution enhancement of DHM via synthesizing a larger hologram. a Synthetic aperture of lens-free DHM using an independent reference wave. b Synthetic aperture of lensless DHM¹⁰², reproduced with permission © IOP Publishing. c Synthesized frequency spectrum. d and e, Images of blood cells without and with applying synthetic aperture, respectively. The images in (c−e) are adapted from Ref. 113 ©The Optical Society.

Figs. 7c−e show the experimental results of the lens-free DHM based on hologram synthesis. The synthesized spectrum is shown in Fig. 7c, and the resolution-enhanced image (Fig. 7e) can be obtained from the Fourier transform of the synthesized spectrum. It is clear that the SA image of the blood cells in Fig. 7e has much finer structures than that in the conventional image in Fig. 7d. Furthermore, it was found that the hologram synthesis enhanced the effective NA from 0.13 to 0.75.

The architecture of a standard on-chip DHM^116-119 is shown in Fig. 8a. A partially coherent light source is used for illumination, usually at > 2–3 cm (z₁ distance) above the sample. A semi-transparent sample is placed on top of an image sensor with a typical spacing of < 1 mm (z₂ distance). As a result, the sample casts an inline hologram, which is directly recorded by a CMOS or CCD image sensor, without using any imaging lenses. The magnification time of the on-chip DHM is defined as M = (z₁ + z₂)/z₁. For z₁ much larger than z₂, the magnification is approximately 1. The hologram recorded in the on-chip DHM is an in-line hologram (Gabor). The direct reconstruction of object-related holographic information from an inline hologram is often obscured by the twin image and the DC term. Phase retrieval approaches using an iterative approach⁷ or additional constraints, such as multiple measurements at different heights^120-123 or illumination angles¹²⁴, can be employed to eliminate twin image noise. There are several factors that can limit the resolution of on-chip DHM, including diffraction, pixel size, image chip area, and coherence of the light source. In practice, the primary limitation of lens-free on-chip DHM resolution comes from the pixel size of the sensor (typically 1–2 μm).

Fig. 8  Pixel super-resolution (PSR) for on-chip lensless DHM. Subpixel shifts in a digital hologram can be achieved through a source shifting, b an array of static light sources, e.g., fiber-coupled LEDs that are individually controllable, sensor shifting, or c sample shifting using e.g., a micro-fluidic flow. d An in-line hologram before PSR. e Pixel super-resolved hologram of the same object. f Reconstructed image of the grating with 338 nm line-width using PSR. g Microscope comparison of the same region of interest. The image is adopted from Ref. 118.

To improve the resolution of on-chip DHM, pixel super-resolution (Pixel SR)^{125, 126} can be adopted by shifting the hologram along the x- and y-directions in subpixel increments. The relative lateral shift of the hologram with respect to the sensor array can also be performed via shifting of the image sensor¹²⁷, the illumination source¹²⁸ (Fig. 8a, b), or the sample¹²⁹ (Fig. 8c), respectively. A low-resolution hologram is captured at each location of the shifted grid. Then, using multiple LR holograms, a higher-resolution hologram can be digitally synthesized by using a “shift-and-add” algorithm, in which the low-resolution (LR) holograms are upsampled, shifted, and digitally added. Alternatively, iterative methods can be used to upsample a high-resolution image from a series of low-resolution images. For this purpose, a cost function can be used for optimization¹²⁵:

This equation contains two operations, namely, norm and regulation, where i represents the recording times. y_i is a low-resolution hologram, and x is the desired high-resolution image. W_i represents the digital processes of shifting and down-sampling of an image. This synthesis method is equivalent to recording holograms with smaller pixel sizes and higher spatial density detectors. The second part γ(x) is a regularization parameter to maintain/regulate the desired quality in the reconstructed image. By solving this optimization function, an image with a pixel size smaller than that of the digital camera can be obtained by synthesizing a series of sub-pixel-shifted LR images. A comparison of Fig. 8d, e shows that PSR can provide a more refined hologram with high-frequency fringes. PSR yields a high-resolution image (Fig. 8f) of a grating (line-width ~300 nm), whose resolution is equivalent to an effective NA~0.9, as shown in Fig. 8g¹¹⁶. Using the PSR techniques, sub-micrometer resolution over a wide FOV of 20–30 mm² can be obtained, providing gigapixel throughput with a simple, compact on-chip DHM design^{130, 131}.

Instead of physically moving the image sensor, the light source, or the sample, virtual pinhole scanning completed by deconvolving a hologram with the measured PSF is an alternative way to improve the resolution of an on-chip DHM with an extended light source¹³². PSR can also be performed by varying the propagation distances (i.e., z₂)¹³³, or by varying the illumination wavelengths in small increments (e.g., 2–3 nm)¹³⁴. To further speed up the sub-hologram recording process by varying the illumination wavelength, color LR holograms can be recorded, where the holograms of the red, reen, and blue illuminations are multiplexed using a sub-pixel shifted Bayer color filter array¹¹⁹. Considering that PSR can only enhance the effective NA of the on-chip DHM to 0.9, a synthetic aperture-based on-chip microscope with which the illumination angle is scanned across the surface of a dome (from −50° to 50°) was proposed, which further increased the effective NA to 1.4, achieving a 250-nm resolution at a wavelength of 700-nm across the very large FOV of 20.5 mm²¹²⁴.

Theoretically, an effective NA of 1+n can be achieved in a lensless on-chip DHM using the PSR/SA approach, assuming that the medium between the source and the sample planes is air. Here, n denotes the refractive index of the medium that fills the space between the sample and sensor planes. However, none of these techniques can be considered to exceed the diffraction limit of light, as they are entirely based on propagating waves that result from coherent light-matter interactions.

Deep-learning^{135, 136} has been demonstrated as being a powerful tool for solving various inverse problems by training a network with a large quantity of paired images. Deep learning can map the relationship between the input and target output distributions without any prior knowledge of the imaging model. Deep learning has been widely used for holographic image reconstruction^137-139, auto-focusing^{140, 141}, and resolution enhancement^{142, 143}.

The deep-learning configuration for the resolution enhancement of DHM is shown in Fig. 9. By training a set of matched LR and high-resolution images, the relationships between them can be learned. Thus, a high resolution can be retrieved from one or multiple low-resolution images captured by a similar setup. The establishment of the mapping function between the high-resolution output image and the LR input images is called the objective function and is expressed as follows:

Fig. 9  Schematics of the deep-learning based method for resolution enhancement of DHM. Images were taken from Ref. 144.

where R_learn is the optimal solution of the weight parameter of the network, x and y are the training set, F¹⁴⁵ is the loss function for appropriate error metrics, and g(θ) is a regularization term for the parameter θ to avoid overfitting¹⁴⁶. By continuously adjusting θ, the error between the output R_θ(y_n) of the neural network and the target image x_n is minimized. Therefore, as long as the establishment of the model is completed through data statistics, it can be applied to solve inverse problems in imaging. Using the network described in Eq. 6, deep learning has also been used to enhance the resolution of data-driven models for the following two systems: (1) lensless DHM and (2) lens-based DHM systems.

In the first case, a deep learning algorithm based on a generative adversarial network (GAN) must be used in an on-chip lensless DHM to achieve sub-pixel resolution¹⁴⁷. In this method, the LR reconstructed images are used as inputs and the corresponding high-resolution images of different samples obtained by the “shift-and-add” approach are used as the labels for training the network. A well-trained framework can generate high-resolution images from unseen LR data without an iterative engine and prior knowledge of the system. In 2019, Liu et al. demonstrated that a resolution-improved image could be obtained from much fewer (e.g., 1 or 4 instead of 36 in conventional Pixel SR algorithm) low-resolution sub-pixel-shifted images of the same object by using a neural network, which significantly reduced both the number of hologram measurements and the computation time necessary for reconstruction¹⁴⁴.

The network training process for the lens-based microscopic imaging system is shown in Fig. 8. The reconstructed images of the holograms obtained under the low-magnification objective were used as inputs, and the counterparts obtained under the high-magnification objective were used as labels in the network to generate an image with a resolution of the high-magnification objective, but with the same FOV as of the low-magnification objective ¹⁴⁴. In this way, the diffraction limit defined by the NA of the objective and the SBP of the system can be broken, so that the holographic reconstruction image has a higher resolution and the reconstruction efficiency is improved as compared with the iterative process for each image. A deep-learning-based upscaling method was proposed for overcoming the trade-off between a large FOV and a high resolution, yielding an enlarged hologram with a smaller effective pixel size (enhanced sample details), for which inline holograms are used as training material for the convolutional neural network¹⁴⁸. In this method, the deep-learning method only uses a single image obtained with a low-NA objective for magnification, allowing detailed analysis of a sample with a high resolution and a large FOV. However, considering that LR-input images do not contain high-resolution components physically, the resolution enhancement capability (or the right guessing of unknown details) of the network is limited by the adequacy of the training data¹⁴⁹. As a remedy, people tend to integrate deep-learning-based strategies and illumination-modulation techniques to enhance the resolution of DHM. For instance, in 2021, Meng et al. demonstrated that the deep-learning algorithm simplifies⁹¹ not only the resolution-enhanced reconstruction process of SI-DHM but also minimizes the recording requirements⁹¹.

In general, for both lensless DHM and lens-based DHM, a deep-learning-based framework can enlarge the SBP of coherent imaging systems using image data and convolutional neural networks, and provides a rapid, non-iterative method for solving inverse image reconstruction or enhancement problems in optics. Furthermore, it has a faster imaging efficiency, even with little prior knowledge¹⁴⁷, and can reconstruct in real time¹⁵⁰. This is because deep-learning methods use large-scale datasets to impose potential constraints on the resolution enhancement problem. A neural network has learned millions of weight parameters from a large amount of training data. These parameters can establish a high-dimensional nonlinear relationship between the input and output, which is difficult to express through simple formulas. Although classical principled algorithms are often outperformed by deep learning models, they also retain key advantages. First, classical algorithms inherently produce outputs that are consistent with their inputs because they rely exclusively on the underlying physics and principles formulated. Second, classic algorithms can be generalized for any valid measurement because they are not limited by the adequacy of the training data. Third, classic algorithms, in contrast to deep learning models, do not produce widely erratic outputs after minute tweaks to their inputs.¹⁴⁹. In light of the above analysis, it is our opinion that deep learning results cannot be blindly trusted, something certainly true of any singular piece of evidence. Incorporating physical models into deep neural networks provides a new idea for expanding the generality and reliability of imaging¹⁵¹, which should be a new trend in deep learning-based resolution enhancement.

In this review, we summarize resolution enhancement approaches for digital holographic microscopy (DHM). Generally, by trading with other degrees of freedom, the spatial resolution is enhanced, and a large field of view (FOV) is preserved (Table 1). Resolution enhancement approaches can be classified into two categories, namely, lens-based DHM and lens-free DHM, based on whether an objective lens is used. In lens-based DHM, oblique illumination, structured illumination, and speckle illumination were used to enhance the spatial resolution of DHM. In sample-distant lens-free DHM, self-extrapolation of holograms can enhance the physical-aperture-limited spatial resolution. Furthermore, the resolution can be further enhanced by synthesizing a larger hologram by scanning the sample, camera, illumination, or inserting a diffraction grating between the sample and the camera. With a lens-free on-chip DHM, resolution enhancement can be obtained by pixel super-resolution and a synthetic aperture by shifting the illumination beam. Recently, deep learning has also been used to enhance the resolution in DHM. Being trained with plenty experimental datasets obtained under well-controlled imaging conditions, deep neural networks enable the enhancement of the spatial resolution or the SBP of DHM in a rapid, non-iterative manner. It is worth pointing out that the resolution enhancement approaches in DHM can surpass the resolution limited by the geometry (NA) of the imaging system in terms of 0.82λ/NA. However, they cannot surpass the physical diffraction limit λ(2NA)¹⁵² in far-field imaging.

In the past decade, intensive efforts have been made to achieve coherent imaging resolution beyond the diffraction limit. There are two coherent imaging approaches that can achieve super-resolution capability. The first is synthetic-aperture DHM using evanescent waves, which can exceed the diffraction limit in the near-field region. The second is super-oscillation illumination or super-oscillatory lens-based optical microscopy¹⁵³. We envisage that there will be an increasing number of investigations concerning the physics of DHM resolution enhancement, and these investigations will widen the applications of DHM in different fields.

The authors acknowledge the support from the National Key Research and Development Program of China (2021YFF0700300), the National Natural Science Foundation of China (NSFC 62075177, 62175112), the Natural Science Foundation of Shaanxi Province (2020JM-193 and 2020JQ-324), and the Fundamental Research Funds for the Central Universities (XJS210503, XJS210504, JC2112, and JB210513).