Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
The Synthesis of Materials with a Hierarchical Structure Based on Tin Dioxide
Next Article in Special Issue
Mo2TiAlC2 as a Saturable Absorber for a Passively Q-Switched Tm:YAlO3 Laser
Previous Article in Journal
Skin Sensitization Potential of Sensitizers in the Presence of Metal Oxide Nanoparticles In Vitro
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Nanomaterials
Volume 14
Issue 22
10.3390/nano14221812
nanomaterials-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Metasurface-Based Image Classification Using Diffractive Deep Neural Network

by
Kaiyang Cheng
1,
Cong Deng
1,
Fengyu Ye
1,
Hongqiang Li
2,3,
Fei Shen
1,*,
Yuancheng Fan
4,* and
Yubin Gong
1,5
1
International School of Microelectronics, Dongguan University of Technology, Dongguan 523808, China
2
College of Electronic and Information Engineering, Tongji University, Shanghai 200092, China
3
The Institute of Dongguan Tongji University, Dongguan 523808, China
4
Key Laboratory of Light Field Manipulation and Information Acquisition, Ministry of Industry and Information Technology and School of Physical Science and Technology, Northwestern Polytechnical University, Xi’an 710129, China
5
National Key Laboratory on Vacuum Electronics, University of Electronic Science and Technology of China (UESTC), Chengdu 610054, China
*
Authors to whom correspondence should be addressed.
Nanomaterials 2024, 14(22), 1812; https://doi.org/10.3390/nano14221812
Submission received: 9 October 2024 / Revised: 8 November 2024 / Accepted: 9 November 2024 / Published: 12 November 2024
(This article belongs to the Special Issue Linear and Nonlinear Optical Properties of Nanomaterials)
Browse Figures
Figure 1
<p>Schematic diagram of the proposed image classifier. An incident plane wave passes through the input plane carrying the information of handwriting digits and undergoes three-layer phased transmitarray which provide layer-by-layer phase modulation with spacing of <span class="html-italic">d</span> = 6.5 μm. After propagating the entire system, light will be focused at a specific location of the output plane corresponding to different digits. From the light intensity distribution pattern, it can be inferred that the target being measured at this moment is the handwritten digit ‘2’.</p> "> Figure 2
<p>Numerical simulation results of D<sup>2</sup>NN. (<b>a</b>) Phase distribution of the hidden layers obtained by the training results of D<sup>2</sup>NN model. (<b>b</b>) Confusion matrix of the classification task on D<sup>2</sup>NN model in the final numerical simulation, achieving an overall accuracy of approximately 96.2% for six-class digit image classification. (<b>c</b>) The evolution of accuracy and loss value during the classification process with increasing the numbers of iteration.</p> "> Figure 3
<p>The geometry structure of unit cell of all-dielectric phased transmitarray. (<b>a</b>) Schematic diagram of a tunable transmissive meta-atom in an all-dielectric Huygens phase. Here, <span class="html-italic">H</span> = 180 nm, <span class="html-italic">h</span> = 350 nm, <span class="html-italic">p</span> = 360 nm. (<b>b</b>) A complete phase coverage from 0 to 2π for the cylinder with radius varies from 30 to 80 nm.</p> "> Figure 4
<p>Schematic diagram of electromagnetic simulation samples and results. (<b>a</b>–<b>f</b>) Selected simulation test sample images. (<b>g</b>–<b>l</b>) Light field distribution images of simulation output planes. (<b>m</b>–<b>r</b>) Energy distribution graphs within each numerical label area.</p> "> Figure 5
<p>Illustrative diagrams of numerical simulation results for animal image samples. (<b>a</b>) The original image of samples marked as “horse”, the converted grayscale image, the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane. (<b>b</b>) The original image of samples marked as “frog”, the converted grayscale image, and the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane.</p> ">
Versions Notes

Abstract

:
The computer-assisted inverse design of photonic computing, especially by leveraging artificial intelligence algorithms, offers great convenience to accelerate the speed of development and improve calculation accuracy. However, traditional thickness-based modulation methods are hindered by large volume and difficult fabrication process, making it hard to meet the data-driven requirements of flexible light modulation. Here, we propose a diffractive deep neural network (D2NN) framework based on a three-layer all-dielectric phased transmitarray as hidden layers, which can perform the classification of handwritten digits. By tailoring the radius of a silicon nanodisk of a meta-atom, the metasurface can realize the phase profile calculated by D2NN and maintain a relative high transmittance of 0.9 at a wavelength of 600 nm. The designed image classifier consists of three layers of phase-only metasurfaces, each of which contains 1024 units, mimicking a fully connected neural network through the diffraction of light fields. The classification task of handwriting digits from the ‘0’ to ‘5’ dataset is verified, with an accuracy of over 90% on the blind test dataset, as well as demonstrated by the full-wave simulation. Furthermore, the performance of the more complex animal image classification task is also validated by increasing the number of neurons to enhance the connectivity of the neural network. This study may provide a possible solution for practical applications such as biomedical detection, image processing, and machine vision based on all-optical computing.

1. Introduction

Analog optical computing [1,2,3,4,5,6], as an alternative computing architecture, has shown great potential with its inherent parallelism, energy efficiency, and low latency. Compared with digital systems, the analog approaches are particularly capable of dealing with massive computational tasks, for instance, in the fields of biological/medical imaging, object classification, astronomical observation, and machine vision [1,7,8,9,10,11]. Despite its potential for various applications, designing a computational optical network is intricate and time-consuming, since the designer has to meticulously select the parameters of the whole structure and nonlinear relationship, making it less straightforward and more unpredictable to find the optimal solution. Recently, with significant advances in computational power, it has become feasible to automatically design optical computing devices based on deep neural network algorithms [12,13]. These advancements allow researchers to leverage the powerful capabilities of deep learning [14,15] to model and optimize optical systems in a more efficient way.
The computer-assisted inverse design of optical devices is an active field [16,17,18,19]; here, we focus on an all-optical diffractive deep neural network (D2NN) framework [20,21,22,23,24,25], where the connections of neurons are formed by the physical reality of the propagation of light through different diffractive optical elements (DOEs). By using conventional deep learning methods such as gradient descent and error back-propagation, the phase and/or amplitude of each optical element can be determined according to the Huygens–Fresnel principle. Nevertheless, in order to achieve the arbitrary and precise modulation capabilities calculated by artificial intelligence, conventional thickness-based modulators cannot meet the requirements as they suffer the drawbacks of being bulky, inefficient, and having low resolution. To overcome these challenges, a kind of two-dimensional artificial electromagnetic (EM) material called a metasurface [26,27,28,29,30,31,32] has been introduced as a powerful tool which can enable us to engineer the spatial distribution of amplitude, phase, and polarization response with subwavelength resolution. Their ability to impart abrupt phase shifts on a scattered wave allow for the creation of tailored wavefronts and the realization of various functionalities such as negative refraction, flat lenses [33,34,35], EM wave absorber, and a signal processing system.
Table 1 shows the computational performance of different D2NN models. As shown in Table 1, diffractive optical neural networks (DONNs [36]) and multi-wavelength diffractive deep neural networks (Multi-wavelength D2NNs [37]) achieve high accuracy on the MNIST dataset using the single-wavelength and multi-wavelength approaches, respectively. The single-layer diffractive neural network (SL-DNN [38]) model improved the expressive capability of single-layer structures by controlling the Fresnel number, resulting in higher accuracy. The diffractive deep neural networks at visible wavelengths (Visible Light D2NN [39]) model introduced a general theory that included a correction formula, quantifying the match between theoretical design and manufacturing system performance (wavelength, neuron size, and manufacturing constraints). The reconfigurable digital all-optical diffractive neural network (R-ODNN [40]) model proposed a reconfigurable structure, enabling the network to be reconfigurable, digital, and non-volatile. In this article, based on the findings of other researchers, in order to ensure an accuracy rate of over 90% while reducing the overall size of the D2NN structure, we propose a metasurface based image classifier using a diffractive neural network algorithm working at wavelength of 600 nm. Three all-dielectric phased transmitarrays [41,42,43,44] are cascaded as hidden layers, where the phase distributions of the metasurfaces are calculated by phase-only D2NN [45] through training a large number of samples in the handwritten digit database. The incident light passing through the metal mask will carry image information and eventually be focused at a specific location after being modulated by the multi-layer scatterers, simulating the fully connected neural network. By adjusting the radius of the silicon nanorods, the meta-atom can achieve a complete 2π phase shift coverage range and maintain high transmittance at a wavelength of 600 nm. Through numerical calculation and verified by the full wave simulation, it is shown that the proposed metasurface image classifier can effectively control the phase of light and achieve an overall accuracy of 96.2% for six-class digit image classification. We believe the proposed system is a potential candidate in fields such as optical device design and defect detection.

2. Materials and Methods

The proposed three-layer phase-only D2NN architecture is illustrated in Figure 1. The samples used for learning and training are derived from the MNIST dataset with handwritten digits from ‘0’ to ‘5’. Unlike traditional artificial neural networks, which can directly extract pixel information from the image samples, here, in order to facilitate the optical implementation, we make some adjustments to the original samples where the grayscale values of digit images are binarized into 0 and 1 with a threshold of 0.2. Pixel values of 0 and 1 indicate the transparent and opaque statuses, respectively, which can be realized by placing a perforated metal plate at the input plane. Then, we pad zeros to the original images (28 × 28 pixels) in order to make it consistent with the pixel size of the diffraction layers (32 × 32 pixels). Each diffraction layer consists of 32 × 32 meta-atoms (corresponding to neurons in D2NN) with an interlayer spacing of d = 6.5 μm.

2.1. Algorithm of D2NN

During the training of the diffractive neural network model, data enter the network through the input layer and propagate forward to produce an output. The weights of the neurons are optimized through an error backpropagation algorithm. According to the Rayleigh–Sommerfeld diffraction formula [46], each point on the diffraction surface can be regarded as a secondary spherical wavelet source, and the spatial distribution of a light field is the result of a coherent superposition of the wavelet sources after diffraction propagation. The diffraction propagation formula for the neurons is expressed as follows:
w i l ( x , y , z ) = z − z i r 2 ( 1 2 π r + 1 j λ ) e j 2 π r λ r = ( x − x i ) 2 + ( y − y i ) 2 + ( z − z i ) 2
where l indicates the l-th layer of the diffractive neural network, i represents the i-th neuron of the l-th layer, r denotes the Euclidean distance between the i-th neuron of the l-th layer and the (l + 1)-th layer, j is the imaginary unit − 1 , λ represents the wavelength of the incoming wave, and w i l ( x , y , z ) is the complex-valued field generated at point ( x , y , z ) of an neuron in layer (l + 1)-th by the i-th neuron located at point ( x i , y i , z i ) in layer l-th. The input plane is designated as 0-th layer. As a secondary wave source, the amplitude and relative phase of the secondary waves generated by the neuron are determined by the incident wave and the transmission function of the neuron. The output light fields of the i-th neurons in the l-th layer can be expressed as follows:
Y i l ( x , y , z ) = w i l ( x , y , z ) ⋅ t i l ( x i , y i , z i ) ⋅ ∑ k Y k l − 1 ( x i , y i , z i ) = w i l ( x , y , z ) ⋅ A ⋅ e j △ θ
where t i l is the transmission function of the neuron, and the input at the i-th neuron of the l-th layer is the superposition of all neurons’ outputs from the previous (l-1)-th layer, which is denoted as ∑ k Y k l − 1 ( x i , y i , z i ) . A represents the output amplitude of the i-th neuron, and Δθ denotes the phase delay produced by the input light wave at the i-th neuron. The transmission coefficient of light can be expressed as follows:
t i l ( x i , y i , z i ) = a i l ( x i , y i , z i ) e j ϕ i l ( x i , y i , z i )
where a i l ( x i , y i , z i ) indicates the neuron’s amplitude, and ϕ i l ( x i , y i , z i ) represents the neuron’s phase. In practical fabrication, the chosen substrate is often optically transparent with constant transmittance. The neurons processed on such a substrate have a constant amplitude, and this type of D2NN is referred to as phase-only D2NN.
However, training with the aforementioned integral formula consumes significant computational time. By introducing plane wave angular spectrum [47] and fast Fourier transform algorithms [48] during the training process, the computation in the spatial domain can be converted to the frequency domain, significantly reducing the computational load and thereby decreasing training time. In this paper, D2NN architecture is modeled using Python (v3.8.18) and the Pytorch (v1.8.1) framework. It runs on a server equipped with an NVIDIA A40 GPU, an AMD EPYC 7513 @2.6 GHz CPU, and 1024 GB RAM, operating on Windows 10 (Microsoft, Redmond, WA, USA). In our numerical analyses, firstly, it is necessary to assign an initial phase distribution to each diffraction layer. We employed the Xavier initialization method [49], which is a prevalent technique for neural network weight initialization. This method effectively prevents issues such as vanishing and exploding gradients by ensuring that the outputs at each layer maintain an appropriate variance, thereby enhancing the convergence speed during training and the generalization capability of the network. Next, we use the mean squared error function as the loss function, which is commonly utilized in D2NN algorithm. The training batch size is set to 16, with a learning rate of 0.001. Each layer comprises 32 × 32 neurons, and the phase parameters of each neuron are iteratively updated using the stochastic gradient descent algorithm to minimize the loss function. It should be noted that the nonlinear function sigmoid 2 π / ( 1 + e − ϕ ) is used to keep the phase ϕ of each update between 0 and 2π during the training.
The D2NN is trained using a dataset of 36,000 samples, with an additional 6030 samples reserved for testing. After that, we conduct the training of 500 epochs for the entirely phase-only three-layer D2NN to ensure sufficient learning and convergence of the network. Figure 2a illustrates the learned phase profile of three diffraction layers, while Figure 2b demonstrates the confusion matrix for the D2NN model, achieving an overall accuracy rate of approximately 96.2% in classifying six types of digit patterns. Figure 2c depicts the changes in loss value and accuracy during the training process; it is shown that the convergence speed of the model is faster in the training process, which indicates that the model training has been effectively optimized and adjusted.

2.2. All-Dielectric Metasurface Design

To perform a proof-of-concept demonstration of our diffractive image classifier framework, we designed a three-layer phased transmitarray to meet the requirement learned by D2NN algorithm. The meta-atom used to simulate neurons in the deep learning algorithm is shown in Figure 3a. It is composed of a silicon dioxide substrate with a periodicity of p = 360 nm and height H = 180 nm, and a cylindrical silicon nanodisk of height h = 350 nm is placed on top of it. Each metasurface consists of 1024 unit cells, each of which has a size of 360 nm × 360 nm × 530 nm. Figure 3b illustrates a complete phase coverage from 0 to 2π for a cylinder with the radius varying from 30 to 80 nm at a wavelength of 600 nm. For simplification in subsequent simulations, the continuous phase distribution from 0 to 2π was discretized into 32 points, each with a fixed phase difference of 11.25° from the neighboring locations [50]. The 32 discrete points are marked as star shapes on the graph, while the spherical markers indicate the amplitude of the unit cell corresponding to these 32 different radii, all of which are nearly greater than 0.9. Within the parameter range of this structure, the size of the silicon cylinder is smaller than the wavelength of the incident light, so the scattering of light is mainly dominated by the properties of a single scatterer and is not significantly affected by coupling between scatterers. At the same time, the design employs gradually varying scatterer sizes to ensure that the size changes between the adjacent scatterers are smooth, thereby avoiding abrupt changes and reducing scattering and diffraction losses of light. This local scattering effect and size design approach make light transmission and phase modulation more efficient, supporting the achievement of high transmittance [41,51].

3. Results

3.1. Digital Images Classification

To verify the role of the metasurface composed of the unit cell as a diffraction layer in the D2NN, the physical model was built in CST Studio Suite based on the conversion of the phase parameters of the training model into the structural parameters of the nanodisk. It was also necessary to properly set parameters such as frequency, plane wave, and boundary conditions. It took about 12 h to obtain the simulation results for the distribution of the output plane field for each sample. Figure 4a–f are the digital samples used for simulation, while Figure 4g–l depict the light intensity distribution on the detector plane for these test samples after electromagnetic simulation. On the detection plane, we arranged six 2 × 2 pixels as the target regions (the red rectangles in figures) corresponding to the six digits [52]. Figure 4m–r show the energy proportion within these six red rectangles. For every test sample, after passing through the three-layer D2NN, there was a target area within these six red rectangles that had the highest energy relative to the others, and the number represented by this area matched the corresponding label category in the algorithm. This demonstrates that the D2NN structure can accurately classify different handwritten digits with a processing speed close to the speed of light.
From the EM simulation results, it is shown that apart from the digit ‘3’, other digits had noticeable focal points in the target areas. It may have arisen from the relatively limited number of meta-atoms we used in the model since the total number of grids will increase exponentially with the increase in neurons, leading to a decrease in the overall network connectivity. This restriction reduces the network’s capacity to extract and convey image features precisely. By increasing the number of neurons in each layer and the number of diffraction layers, nonlinear functions can be more suitable for diffractive neural networks, thereby improving the recognition accuracy and recognizing more complex images. Alternatively, achieving fully connected topologies between the layers by optimizing the spacing of diffraction layers can ensure comprehensive neuronal diffraction interconnectivity thus guaranteeing effective information transmission between layers. From the perspective of a metasurface, one can choose a meta-atom capable of modulating both phase and amplitude simultaneously to achieve a more precise classification.

3.2. Animal Images Classification

To study the applications on complex graphs, we also attempted to use the D2NN to identify and classify animal images. The samples came from the CIFAR-10 dataset; we selected six types of animals (including birds, cats, deer, dogs, frogs, and horses) to train the D2NN model. The dataset comprised three-channel RGB images, since D2NN is designed to handle monochromatic light, and we converted these images to grayscale during the preprocessing stage thus transforming them into one-channel images. Figure 5a,b show the original images and grayscale images of a horse and a frog, as well as the energy distribution map of the detector plane after D2NN processing. In order to classify those complex images, we expanded the size and increased the number of diffraction layers to 128 × 128 pixels and five layers, respectively, to enhance the computational capability of the D2NN. The numerical results show that more complex animal image classification can be successfully achieved. Due to the considerable noise and complexity of features in the images, the classification accuracy achieved by the model was relatively low for numerical images at only 55.1%. Next, we plan to integrate the optical 4f system and make improvements to the image preprocessing steps. Our aim is to better extract the image features fed into the D2NN and minimize noise interference thereby enhancing overall performance.

4. Conclusions

This research mainly demonstrates the design, numerical simulation, and full-wave electromagnetic validation of a phase-only all-optical D2NN based on an all-dielectric phased transmitarray, specifically in the performance of handwritten digit classification tasks. The detailed design process for the metasurface, along with the related numerical models and algorithms, is provided. The classification accuracy in numerical simulations reached about 90%. Selecting some digital samples from the dataset for full-wave electromagnetic simulations, which are consistent with the numerical simulation results, demonstrated the feasibility of D2NN in applications at the frequency of 600 nm.

Author Contributions

Conceptualization, K.C.; Methodology, C.D. and F.Y.; Software, C.D.; Supervision, Y.F.; Validation, H.L., F.S. and Y.G.; Visualization, C.D.; Writing—Original Draft, C.D. and F.Y.; Writing—Review and Editing, K.C. and Y.F. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Guangdong Basic and Applied Basic Research Foundation (2021A1515110733, 2022B1515130011, 2021B1515140049), Dongguan Science and Technology Commissioner Project (20231800500212), Guangdong Provincial Key Laboratory Project (High-Performance Integrated Circuits and Systems Laboratory) grant number No. 2023KSYS003, National Key R&D Program of China (Grant No. 2023YFB3811400).

Data Availability Statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding authors.

Acknowledgments

The authors thank Luo Xu Hao for their excellent assistance with the diffraction algorithm.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Abdollahramezani, S.; Hemmatyar, O.; Adibi, A. Meta-optics for spatial optical analog computing. J Nanophotonics. Nanophotonics 2020, 9, 4075–4095. [Google Scholar] [CrossRef]
  2. Zangeneh-Nejad, F.; Sounas, D.L.; Alù, A.; Fleury, R. Analogue computing with metamaterials. Nat. Rev. Mater. 2020, 6, 207–225. [Google Scholar] [CrossRef]
  3. Zhou, Y.; Wu, W.; Chen, R.; Chen, W.; Chen, R.; Ma, Y. Analog Optical Spatial Differentiators Based on Dielectric Metasurfaces. Adv. Opt. Mater. 2020, 8, 1901523. [Google Scholar] [CrossRef]
  4. Zhou, Y.; Zheng, H.; Kravchenko, I.I.; Valentine, J. Flat optics for image differentiation. Nat. Photonics 2020, 14, 316–323. [Google Scholar] [CrossRef]
  5. Goh, H.; Alù, A. Nonlocal Scatterer for Compact Wave-Based Analog Computing. Phys. Rev. Lett. 2022, 128, 073201. [Google Scholar] [CrossRef]
  6. Chen, Y.; Nazhamaiti, M.; Xu, H.; Meng, Y.; Zhou, T.; Li, G.; Fan, J.; Wei, Q.; Wu, J.; Qiao, F.; et al. All-analog photoelectronic chip for high-speed vision tasks. Nature 2023, 623, 48–57. [Google Scholar] [CrossRef]
  7. Kam, Z.; Hanser, B.; Gustafsson, M.; Agard, D.; Sedat, J. Computational adaptive optics for live three-dimensional biological imaging. Proc. Natl. Acad. Sci. USA 2001, 98, 3790–3795. [Google Scholar] [CrossRef]
  8. Jia, P.; Zhao, Y.; Xue, G.; Cai, D. Optical transient object classification in wide-field small aperture telescopes with a neural network. Astron. J. 2019, 157, 250. [Google Scholar] [CrossRef]
  9. Oppenheimer, B.R.; Hinkley, S. High-contrast observations in optical and infrared astronomy. Annu. Rev. Astron. Astrophys. 2009, 47, 253–289. [Google Scholar] [CrossRef]
  10. Xu, Z.; Yuan, X.; Zhou, T.; Fang, L. A multichannel optical computing architecture for advanced machine vision. Light Sci. Appl. 2022, 11, 255. [Google Scholar] [CrossRef]
  11. Liu, C.; Ma, Q.; Luo, Z.J.; Hong, Q.R.; Xiao, Q.; Zhang, H.C.; Miao, L.; Yu, W.M.; Cheng, Q.; Li, L. A programmable diffractive deep neural network based on a digital-coding metasurface array. Nat. Electron. 2022, 5, 113–122. [Google Scholar] [CrossRef]
  12. Wang, T.; Ma, S.-Y.; Wright, L.G.; Onodera, T.; Richard, B.C.; McMahon, P.L. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 2022, 13, 123. [Google Scholar] [CrossRef] [PubMed]
  13. Ashtiani, F.; Geers, A.J.; Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 2022, 606, 501–506. [Google Scholar] [CrossRef] [PubMed]
  14. Jamshidi, M.B.; Lalbakhsh, A.; Talla, J.; Peroutka, Z.; Roshani, S.; Matousek, V.; Roshani, S.; Mirmozafari, M.; Malek, Z.; La Spada, L. Deep learning techniques and COVID-19 drug discovery: Fundamentals, state-of-the-art and future directions. In Emerging Technologies During the Era of COVID-19 Pandemic; Springer: Cham, Switzerland, 2021; pp. 9–31. [Google Scholar]
  15. Naveed, H.; Khan, A.U.; Qiu, S.; Saqib, M.; Anwar, S.; Usman, M.; Akhtar, N.; Barnes, N.; Mian, A. A comprehensive overview of large language models. arXiv 2023, arXiv:2307.06435. [Google Scholar]
  16. Zhu, C.; Bamidele, E.A.; Shen, X.; Zhu, G.; Li, B. Machine learning aided design and optimization of thermal metamaterials. Chem. Rev. 2024, 124, 4258–4331. [Google Scholar] [CrossRef]
  17. Kojima, K.; Tahersima, M.H.; Koike-Akino, T.; Jha, D.K.; Tang, Y.; Wang, Y.; Parsons, K. Deep neural networks for inverse design of nanophotonic devices. J. Light. Technol. 2021, 39, 1010–1019. [Google Scholar] [CrossRef]
  18. So, S.; Badloe, T.; Noh, J.; Bravo-Abad, J.; Rho, J. Deep learning enabled inverse design in nanophotonics. Nanophotonics 2020, 9, 1041–1057. [Google Scholar] [CrossRef]
  19. Han, X.; Fan, Z.; Liu, Z.; Li, C.; Guo, L.J. Inverse design of metasurface optical filters using deep neural network with high degrees of freedom. InfoMat 2021, 3, 432–442. [Google Scholar] [CrossRef]
  20. Lin, X.; Rivenson, Y.; Yardimei, N.T.; Veli, M.; Luo, Y.; Jarrahi, M.; Ozcan, A. All-optical machine learning using diffractive deep neural networks. Science 2018, 361, 1004. [Google Scholar] [CrossRef]
  21. Yan, T.; Wu, J.; Zhou, T.; Xie, H.; Xu, F.; Fan, J.; Fang, L.; Lin, X.; Dai, Q. Fourier-space Diffractive Deep Neural Network. Phys. Rev. Lett. 2019, 123, 023901. [Google Scholar] [CrossRef]
  22. Bai, B.; Li, Y.; Luo, Y.; Li, X.; Cetintas, E.; Jarrahi, M.; Ozcan, A. All-optical image classification through unknown random diffusers using a single-pixel diffractive network. Light-Sci. Appl. 2023, 12, 69. [Google Scholar] [CrossRef] [PubMed]
  23. Luo, Y.; Mengu, D.; Yardimci, N.T.; Rivenson, Y.; Veli, M.; Jarrahi, M.; Ozcan, A. Design of task-specific optical systems using broadband diffractive neural networks. Light-Sci. Appl. 2019, 8, 112. [Google Scholar] [CrossRef] [PubMed]
  24. Li, J.X.; Mengu, D.; Yardimci, N.T.; Luo, Y.; Li, X.R.; Veli, M.; Rivenson, Y.; Jarrahi, M.; Ozcan, A. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. 2021, 7, eabd7690. [Google Scholar] [CrossRef] [PubMed]
  25. Huang, C.; Sorger, V.J.; Miscuglio, M.; Al-Qadasi, M.; Mukherjee, A.; Lampe, L.; Nichols, M.; Tait, A.N.; Ferreira de Lima, T.; Marquez, B.A.; et al. Prospects and applications of photonic neural networks. Adv. Phys. X 2022, 7, 1981155. [Google Scholar] [CrossRef]
  26. Sun, S.; He, Q.; Hao, J.; Xiao, S.; Zhou, L. Electromagnetic metasurfaces: Physics and applications. Adv. Opt. Photonics 2019, 11, 380–479. [Google Scholar] [CrossRef]
  27. Hail, C.U.; Michel, A.-K.U.; Poulikakos, D.; Eghlidi, H. Optical Metasurfaces: Evolving from Passive to Adaptive. Adv. Opt. Mater. 2019, 7, 1801786. [Google Scholar] [CrossRef]
  28. Luo, X. Subwavelength Artificial Structures: Opening a New Era for Engineering Optics. Adv. Mater. 2019, 31, 1804680. [Google Scholar] [CrossRef]
  29. Luo, X. Metamaterials and Metasurfaces. Adv. Opt. Mater. 2019, 7, 1900885. [Google Scholar] [CrossRef]
  30. Wang, J.; Li, Y.; Jiang, Z.H.; Shi, T.; Tang, M.; Zhou, Z.; Chen, Z.N.; Qiu, C. Metantenna: When Metasurface Meets Antenna Again. IEEE Trans. Antennas Propag. 2020, 68, 1332–1347. [Google Scholar] [CrossRef]
  31. Yang, J.; Gurung, S.; Bej, S.; Ni, P.; Howard Lee, H.W. Active optical metasurfaces: Comprehensive review on physics, mechanisms, and prospective applications. Rep. Prog. Phys. 2022, 85, 036101. [Google Scholar] [CrossRef]
  32. Xu, J.; Yang, R.; Fan, Y.; Fu, Q.; Zhang, F. A review of tunable electromagnetic metamaterials with anisotropic liquid crystals. Front. Phys. 2021, 9, 633104. [Google Scholar] [CrossRef]
  33. Cheng, K.; Wei, Z.; Fan, Y.; Zhang, X.; Wu, C.; Li, H. Realizing broadband transparency via manipulating the hybrid coupling modes in metasurfaces for high-efficiency metalens. Adv. Opt. Mater. 2019, 7, 1900016. [Google Scholar] [CrossRef]
  34. Wang, S.; Wu, P.C.; Su, V.-C.; Lai, Y.-C.; Chen, M.-K.; Kuo, H.Y.; Chen, B.H.; Chen, Y.H.; Huang, T.-T.; Wang, J.-H. A broadband achromatic metalens in the visible. Nat. Nanotechnol. 2018, 13, 227–232. [Google Scholar] [CrossRef] [PubMed]
  35. Li, L.; Zhao, H.; Liu, C.; Li, L.; Cui, T.J. Intelligent metasurfaces: Control, communication and computing. eLight 2022, 2, 7. [Google Scholar] [CrossRef]
  36. Filipovich, M.J.; Malyshev, A.; Lvovsky, A. Role of spatial coherence in diffractive optical neural networks. arXiv 2023, arXiv:2310.03679. [Google Scholar] [CrossRef]
  37. Duan, Z.Y.; Chen, H.; Lin, X. Optical multi-task learning using multi-wavelength diffractive deep neural networks. Nanophotonics 2023, 12, 893–903. [Google Scholar] [CrossRef]
  38. Zheng, M.; Shi, L.; Zi, J. Optimize performance of a diffractive neural network by controlling the Fresnel number. Photonics Res. 2022, 10, 2667–2676. [Google Scholar] [CrossRef]
  39. Chen, H.; Feng, J.A.; Jiang, M.W.; Wang, Y.Q.; Lin, J.; Tan, J.B.; Jin, P. Diffractive Deep Neural Networks at Visible Wavelengths. Engineering 2021, 7, 1483–1491. [Google Scholar] [CrossRef]
  40. Wu, C.; Zhao, J.; Hu, Q.; Zeng, R.; Zhang, M. Non-volatile Reconfigurable Digital Optical Diffractive Neural Network Based on Phase Change Material. arXiv 2023, arXiv:2305.11196. [Google Scholar]
  41. Tanriover, I.; Demir, H.V. Broad-band polarization-insensitive all-dielectric metalens enabled by intentional off-resonance waveguiding at mid-wave infrared. Appl. Phys. Lett. 2019, 114, 043105. [Google Scholar] [CrossRef]
  42. Ollanik, A.J.; Smith, J.A.; Belue, M.J.; Escarra, M.D. High-efficiency all-dielectric Huygens metasurfaces from the ultraviolet to the infrared. ACS Photonics 2018, 5, 1351–1358. [Google Scholar] [CrossRef]
  43. Decker, M.; Staude, I.; Falkner, M.; Dominguez, J.; Neshev, D.N.; Brener, I.; Pertsch, T.; Kivshar, Y.S. High-efficiency dielectric Huygens’ surfaces. Adv. Opt. Mater. 2015, 3, 813–820. [Google Scholar] [CrossRef]
  44. Zhu, W.; Fan, Y.; Yang, R.; Geng, G.; Fu, Q.; Gu, C.; Li, J.; Zhang, F. Polarization-multiplexed silicon metasurfaces for multi-channel visible light modulation. Adv. Funct. Mater. 2022, 32, 2200013. [Google Scholar] [CrossRef]
  45. He, S.S.; Wang, R.S.; Luo, H.L. Computing metasurfaces for all-optical image processing: A brief review. Nanophotonics 2022, 11, 1083–1108. [Google Scholar] [CrossRef]
  46. Goodman, J.W.; Cox, M.E. Introduction to Fourier Optics. Phys. Today 1969, 22, 97–101. [Google Scholar] [CrossRef]
  47. Matsushima, K.; Shimobaba, T. Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields. Opt. Express 2009, 17, 19662–19673. [Google Scholar] [CrossRef]
  48. Brigham, E.O.; Morrow, R. The fast Fourier transform. IEEE Spectr. 1967, 4, 63–70. [Google Scholar] [CrossRef]
  49. Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; JMLR Workshop and Conference Proceedings. pp. 249–256. [Google Scholar]
  50. Gu, Z.; Ma, Q.; Gao, X.; You, J.W.; Cui, T.J. Classification of metal handwritten digits based on microwave diffractive deep neural network. Adv. Opt. Mater. 2024, 12, 2301938. [Google Scholar] [CrossRef]
  51. Arbabi, A.; Horie, Y.; Ball, A.J.; Bagheri, M.; Faraon, A. Subwavelength-thick lenses with high numerical apertures and large efficiency based on high-contrast transmitarrays. Nat. Commun. 2015, 6, 7069. [Google Scholar] [CrossRef]
  52. Luo, X.; Hu, Y.; Ou, X.; Li, X.; Lai, J.; Liu, N.; Cheng, X.; Pan, A.; Duan, H. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light Sci. Appl. 2022, 11, 158. [Google Scholar] [CrossRef]
Nanomaterials 14 01812 g001
Figure 1. Schematic diagram of the proposed image classifier. An incident plane wave passes through the input plane carrying the information of handwriting digits and undergoes three-layer phased transmitarray which provide layer-by-layer phase modulation with spacing of d = 6.5 μm. After propagating the entire system, light will be focused at a specific location of the output plane corresponding to different digits. From the light intensity distribution pattern, it can be inferred that the target being measured at this moment is the handwritten digit ‘2’.
Figure 1. Schematic diagram of the proposed image classifier. An incident plane wave passes through the input plane carrying the information of handwriting digits and undergoes three-layer phased transmitarray which provide layer-by-layer phase modulation with spacing of d = 6.5 μm. After propagating the entire system, light will be focused at a specific location of the output plane corresponding to different digits. From the light intensity distribution pattern, it can be inferred that the target being measured at this moment is the handwritten digit ‘2’.
Nanomaterials 14 01812 g001
Nanomaterials 14 01812 g002
Figure 2. Numerical simulation results of D2NN. (a) Phase distribution of the hidden layers obtained by the training results of D2NN model. (b) Confusion matrix of the classification task on D2NN model in the final numerical simulation, achieving an overall accuracy of approximately 96.2% for six-class digit image classification. (c) The evolution of accuracy and loss value during the classification process with increasing the numbers of iteration.
Figure 2. Numerical simulation results of D2NN. (a) Phase distribution of the hidden layers obtained by the training results of D2NN model. (b) Confusion matrix of the classification task on D2NN model in the final numerical simulation, achieving an overall accuracy of approximately 96.2% for six-class digit image classification. (c) The evolution of accuracy and loss value during the classification process with increasing the numbers of iteration.
Nanomaterials 14 01812 g002
Nanomaterials 14 01812 g003
Figure 3. The geometry structure of unit cell of all-dielectric phased transmitarray. (a) Schematic diagram of a tunable transmissive meta-atom in an all-dielectric Huygens phase. Here, H = 180 nm, h = 350 nm, p = 360 nm. (b) A complete phase coverage from 0 to 2π for the cylinder with radius varies from 30 to 80 nm.
Figure 3. The geometry structure of unit cell of all-dielectric phased transmitarray. (a) Schematic diagram of a tunable transmissive meta-atom in an all-dielectric Huygens phase. Here, H = 180 nm, h = 350 nm, p = 360 nm. (b) A complete phase coverage from 0 to 2π for the cylinder with radius varies from 30 to 80 nm.
Nanomaterials 14 01812 g003
Nanomaterials 14 01812 g004
Figure 4. Schematic diagram of electromagnetic simulation samples and results. (a–f) Selected simulation test sample images. (g–l) Light field distribution images of simulation output planes. (m–r) Energy distribution graphs within each numerical label area.
Figure 4. Schematic diagram of electromagnetic simulation samples and results. (a–f) Selected simulation test sample images. (g–l) Light field distribution images of simulation output planes. (m–r) Energy distribution graphs within each numerical label area.
Nanomaterials 14 01812 g004
Nanomaterials 14 01812 g005
Figure 5. Illustrative diagrams of numerical simulation results for animal image samples. (a) The original image of samples marked as “horse”, the converted grayscale image, the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane. (b) The original image of samples marked as “frog”, the converted grayscale image, and the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane.
Figure 5. Illustrative diagrams of numerical simulation results for animal image samples. (a) The original image of samples marked as “horse”, the converted grayscale image, the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane. (b) The original image of samples marked as “frog”, the converted grayscale image, and the light field distribution on the detector plane through five diffraction layers, and the energy proportion within each marked area on the detector plane.
Nanomaterials 14 01812 g005
Table 1. Computational performance of different D2NN models on MNIST dataset.
Table 1. Computational performance of different D2NN models on MNIST dataset.
ModelNetwork SizePixel SizeIncident WavelengthAccuracy (%)
DONNs [36]100 × 100 × 510 μm × 10 μm700 nm97.54
Multi-wavelength D2NNs [37]200 × 200 × 54 μm × 4 μm400 nm, 500 nm and 700 nm95.6
SL-DNN [38]200 × 200 × 18 μm × 8 μm515 nm97.08
Visible
Light D2NN [39]		1000 × 1000 × 54 μm × 4 μm632 nm91.57
R-ODNN [40]120 × 120 × 31 μm × 1 μm1550 nm94.46
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Cheng, K.; Deng, C.; Ye, F.; Li, H.; Shen, F.; Fan, Y.; Gong, Y. Metasurface-Based Image Classification Using Diffractive Deep Neural Network. Nanomaterials 2024, 14, 1812. https://doi.org/10.3390/nano14221812

AMA Style

Cheng K, Deng C, Ye F, Li H, Shen F, Fan Y, Gong Y. Metasurface-Based Image Classification Using Diffractive Deep Neural Network. Nanomaterials. 2024; 14(22):1812. https://doi.org/10.3390/nano14221812

Chicago/Turabian Style

Cheng, Kaiyang, Cong Deng, Fengyu Ye, Hongqiang Li, Fei Shen, Yuancheng Fan, and Yubin Gong. 2024. "Metasurface-Based Image Classification Using Diffractive Deep Neural Network" Nanomaterials 14, no. 22: 1812. https://doi.org/10.3390/nano14221812

APA Style

Cheng, K., Deng, C., Ye, F., Li, H., Shen, F., Fan, Y., & Gong, Y. (2024). Metasurface-Based Image Classification Using Diffractive Deep Neural Network. Nanomaterials, 14(22), 1812. https://doi.org/10.3390/nano14221812

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop