research-article

Open access

Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Two-Photon Neural Lithography Simulator

Authors:

Guangyuan Zhao,

Peter SoAuthors Info & Claims

SA '23: SIGGRAPH Asia 2023 Conference Papers

Article No.: 56, Pages 1 - 9

https://doi.org/10.1145/3610548.3618251

Published: 11 December 2023 Publication History

All formats PDF

Abstract

We introduce neural lithography to address the ‘design-to-manufacturing’ gap in computational optics. Computational optics with large design degrees of freedom enable advanced functionalities and performance beyond traditional optics. However, the existing design approaches often overlook the numerical modeling of the manufacturing process, which can result in significant performance deviation between the design and the fabricated optics. To bridge this gap, we, for the first time, propose a fully differentiable design framework that integrates a pre-trained photolithography simulator into the model-based optical design loop. Leveraging a blend of physics-informed modeling and data-driven training using experimentally collected datasets, our photolithography simulator serves as a regularizer on fabrication feasibility during design, compensating for structure discrepancies introduced in the lithography process. We demonstrate the effectiveness of our approach through two typical tasks in computational optics, where we design and fabricate a holographic optical element (HOE) and a multi-level diffractive lens (MDL) using a two-photon lithography system, showcasing improved optical performance on the task-specific metrics. The source code for this work is available on the project page: https://neural-litho.github.io.

Supplemental Material

PDF File

Supplementary Documents

Download
37.47 MB

References

[1]

Graham Arthur and Brian Martin. 1996. Investigation of photoresist-specific optical proximity effect. Microelectronic engineering 30, 1-4 (1996), 133–136.

[2]

Thomas Auzinger, Wolfgang Heidrich, and Bernd Bickel. 2018. Computational design of nanostructural color for additive manufacturing. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–16.

Digital Library

[3]

Yunhao Ba, Guangyuan Zhao, and Achuta Kadambi. 2019. Blending diverse physical priors with neural networks. arXiv preprint arXiv:1910.00201 (2019).

[4]

Seung-Hwan Baek, Hayato Ikoma, Daniel S Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, and Min H Kim. 2021. Single-shot hyperspectral-depth imaging with learned diffractive optics. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2651–2660.

[5]

Sourangsu Banerji, Monjurul Meem, Apratim Majumder, Fernando Guevara Vasquez, Berardi Sensale-Rodriguez, and Rajesh Menon. 2019. Imaging with flat optics: metalenses or diffractive lenses?Optica 6, 6 (2019), 805–810.

[6]

Steven Barcelo and Zhiyong Li. 2016. Nanoimprint lithography for nanodevice fabrication. Nano Convergence 3, 1 (2016), 1–9.

[7]

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research 18 (2018), 1–43.

Digital Library

[8]

Tomasz Blachowicz, Guido Ehrmann, and Andrea Ehrmann. 2021. Optical elements from 3D printed polymers. e-Polymers 21, 1 (2021), 549–565.

[9]

Vivek Boominathan, Jesse K Adams, Jacob T Robinson, and Ashok Veeraraghavan. 2020. Phlatcam: Designed phase-mask based thin lensless camera. IEEE transactions on pattern analysis and machine intelligence 42, 7 (2020), 1618–1629.

[10]

Thomas Cecil, Danping Peng, Daniel Abrams, Stanley J Osher, and Eli Yablonovitch. 2022. Advances in Inverse Lithography. ACS Photonics (2022).

[11]

Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–18.

Digital Library

[12]

Pierre Chevalier, Patrick Quéméré, Sébastien Bérard-Bergery, Jean-Baptiste Henry, Charlotte Beylier, and Jérôme Vaillant. 2021. Rigorous Model-Based Mask Data Preparation Algorithm Applied to Grayscale Lithography for the Patterning at the Micrometer Scale. Journal of Microelectromechanical Systems 30, 3 (2021), 442–455.

[13]

Nicolas B Cobb, Avideh Zakhor, and Eugene A Miloslavsky. 1996. Mathematical and CAD framework for proximity correction. In Optical Microlithography IX, Vol. 2726. SPIE, 208–222.

[14]

Benoît Colson, Patrice Marcotte, and Gilles Savard. 2007. An overview of bilevel optimization. Annals of operations research 153 (2007), 235–256.

[15]

Stephan Dottermusch, Dmitry Busko, Malte Langenhorst, Ulrich W Paetzold, and Bryce S Richards. 2019. Exposure-dependent refractive index of Nanoscribe IP-Dip photoresist layers. Optics letters 44, 1 (2019), 29–32.

[16]

Sean D’Silva, Thomas Mülders, Hans-Jürgen Stock, and Andreas Erdmann. 2021. Modeling the impact of shrinkage effects on photoresist development. Journal of Micro/Nanopatterning, Materials, and Metrology 20, 1 (2021), 014602–014602.

[17]

Dennis F Gardner, Michael Tanksalvala, Elisabeth R Shanblatt, Xiaoshi Zhang, Benjamin R Galloway, Christina L Porter, Robert Karl Jr, Charles Bevis, Daniel E Adams, Henry C Kapteyn, 2017. Subwavelength coherent imaging of periodic samples using a 13.5 nm tabletop high-harmonic light source. Nature Photonics 11, 4 (2017), 259–263.

[18]

Franz J Giessibl. 2003. Advances in atomic force microscopy. Reviews of modern physics 75, 3 (2003), 949.

[19]

Elena Goi, Steffen Schoenhardt, and Min Gu. 2022. Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks. Nature Communications 13, 1 (2022), 7531.

[20]

Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company publishers.

[21]

Vincent Hahn, Pascal Kiefer, Tobias Frenzel, Jingyuan Qu, Eva Blasco, Christopher Barner-Kowollik, and Martin Wegener. 2020. Rapid assembly of small materials building blocks (voxels) into large functional 3D metamaterials. Advanced Functional Materials 30, 26 (2020), 1907795.

[22]

V Harinarayana and YC Shin. 2021. Two-photon lithography for three-dimensional fabrication in micro/nanoscale regime: A comprehensive review. Optics & Laser Technology 142 (2021), 107180.

[23]

ICRA. 2023. Real2Sim in ICRA 2023. https://real2sim.com/ Accessed: May 14, 2023.

[24]

Hayato Ikoma, Cindy M Nguyen, Christopher A Metzler, Yifan Peng, and Gordon Wetzstein. 2021. Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In 2021 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–12.

[25]

Intel. 2023. Intel® RealSense™ Depth Camera. https://www.intelrealsense.com/depth-camera-d415/ Accessed: May 14, 2023.

[26]

Changwon Jang, Kiseung Bang, Minseok Chae, Byoungho Lee, and Douglas Lanman. 2022. Waveguide Holography: Towards True 3D Holographic Glasses. arXiv preprint arXiv:2211.02784 (2022).

[27]

Changwon Jang, Olivier Mercier, Kiseung Bang, Gang Li, Yang Zhao, and Douglas Lanman. 2020. Design and fabrication of freeform holographic optical elements. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.

Digital Library

[28]

Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016).

[29]

Bentian Jiang, Lixin Liu, Yuzhe Ma, Hang Zhang, Bei Yu, and Evangeline FY Young. 2020. Neural-ILT: Migrating ILT to neural networks for mask printability and complexity co-optimization. In Proceedings of the 39th International Conference on Computer-Aided Design. 1–9.

Digital Library

[30]

E Koleva, K Vutova, B Asparuhova, I Kostic, K Cvetkov, and V Gerasimov. 2018. Modeling approaches for electron beam lithography. In Journal of Physics: Conference Series, Vol. 1089. IOP Publishing, 012016.

[31]

Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone. 2020. High resolution étendue expansion for holographic displays. ACM Transactions on Graphics (TOG) 39, 4 (2020), 66–1.

Digital Library

[32]

Nicolas Lang, Sven Enns, Julian Hering, and Georg von Freymann. 2022. Towards efficient structure prediction and pre-compensation in multi-photon lithography. Optics Express 30, 16 (2022), 28805–28816.

[33]

Kyung Chul Lee, Junghyun Bae, Nakkyu Baek, Jaewoo Jung, Wook Park, and Seung Ah Lee. 2023. Design and single-shot fabrication of lensless cameras with arbitrary point spread functions. Optica 10, 1 (2023), 72–80.

[34]

Anat Levin, Daniel Glasner, Ying Xiong, Frédo Durand, William Freeman, Wojciech Matusik, and Todd Zickler. 2013. Fabricating BRDFs at high spatial resolution using wave optics. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–14.

Digital Library

[35]

Y Li, S Park, M McLamb, M Lata, S Schöche, D Childers, ID Aggarwal, MK Poutous, G Boreman, and Tino Hofmann. 2019. UV to NIR optical properties of IP-Dip, IP-L, and IP-S after two-photon polymerization determined by spectroscopic ellipsometry. Optical Materials Express 9, 11 (2019), 4318–4328.

[36]

Wei-Ping Liao, Hsueh-Li Liu, Yu-Fan Lin, Sheng-Siang Su, Yu-Teng Chen, Guan-Bo Lin, Tsung-Chieh Tseng, Tong-Ke Lin, Chun-Chi Chen, Wen-Hsien Huang, 2022. I-line photolithographic metalenses enabled by distributed optical proximity correction with a deep-learning model. Optics Express 30, 12 (2022), 21184–21194.

[37]

Daniel Christoph Meisel, Marcus Diem, Markus Deubel, Fabián Pérez-Willard, Stefan Linden, Dagmar Gerthsen, Kurt Busch, and Martin Wegener. 2006. Shrinkage precompensation of holographic three-dimensional photonic-crystal templates. advanced materials 18, 22 (2006), 2964–2968.

[38]

Christopher A Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. 2020. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1375–1385.

[39]

Reut Orange-Kedem, Elias Nehme, Lucien E Weiss, Boris Ferdman, Onit Alalouf, Nadav Opatovski, and Yoav Shechtman. 2021. 3D printable diffractive optical elements by liquid immersion. Nature communications 12, 1 (2021), 3067.

[40]

Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural holography with camera-in-the-loop training. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–14.

Digital Library

[41]

Amyn Poonawala and Peyman Milanfar. 2007. Mask design for optical microlithography—an inverse imaging problem. IEEE Transactions on Image Processing 16, 3 (2007), 774–788.

Digital Library

[42]

Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. 2020. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3674–3683.

[43]

Charles Roques-Carmes, Zin Lin, Rasmus E Christiansen, Yannick Salamin, Steven E Kooi, John D Joannopoulos, Steven G Johnson, and Marin Soljacic. 2022. Toward 3D-printed inverse-designed metaoptics. ACS Photonics 9, 1 (2022), 43–51.

[44]

Sourabh K Saha, Chuck Divin, Jefferson A Cuadra, and Robert M Panas. 2017. Effect of proximity of features on the damage threshold during submicron additive manufacturing via two-photon polymerization. Journal of Micro and Nano-Manufacturing 5, 3 (2017).

[45]

Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.

Digital Library

[46]

Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. 2020a. Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1386–1396.

[47]

Qilin Sun, Jian Zhang, Xiong Dun, Bernard Ghanem, Yifan Peng, and Wolfgang Heidrich. 2020b. End-to-end learned, optically coded super-resolution SPAD camera. ACM Transactions on Graphics (TOG) 39, 2 (2020), 1–14.

Digital Library

[48]

Synopsys. 2023. S-Litho: Predictive Modeling of Lithographic Processes. https://www.synopsys.com/silicon/mask-synthesis/sentaurus-lithography.html Accessed: Aug 27, 2023.

[49]

Ahmad P Tafti, Andrew B Kirkpatrick, Zahrasadat Alavi, Heather A Owen, and Zeyun Yu. 2015. Recent advances in 3D SEM surface reconstruction. Micron 78 (2015), 54–66.

[50]

Shiyu Tan, Yicheng Wu, Shoou-I Yu, and Ashok Veeraraghavan. 2021. Codedstereo: Learned phase masks for large depth-of-field stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7170–7179.

[51]

Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar, and Felix Heide. 2021. Neural nano-optics for high-quality thin lens imaging. Nature communications 12, 1 (2021), 6493.

[52]

Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl ST Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. 2019. Hyperparameter optimization in black-box image processing using differentiable proxies.ACM Trans. Graph. 38, 4 (2019), 27–1.

Digital Library

[53]

Ethan Tseng, Yuxuan Zhang, Lars Jebe, Xuaner Zhang, Zhihao Xia, Yifei Fan, Felix Heide, and Jiawen Chen. 2022. Neural Photo-Finishing. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–15.

Digital Library

[54]

Christian Wagner and Noreen Harned. 2010. Lithography gets extreme. Nature Photonics 4, 1 (2010), 24–26.

[55]

Hao Wang, Hongtao Wang, Wang Zhang, and Joel KW Yang. 2020. Toward near-perfect diffractive optical elements via nanoscale 3D printing. ACS nano 14, 8 (2020), 10452–10461.

[56]

Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. 2021. Geometry processing with neural fields. Advances in Neural Information Processing Systems 34 (2021), 22483–22497.

[57]

Haoyu Yang, Shuhe Li, Yuzhe Ma, Bei Yu, and Evangeline FY Young. 2018. GAN-OPC: Mask optimization with lithography-guided generative adversarial nets. In Proceedings of the 55th Annual Design Automation Conference. 1–6.

Digital Library

[58]

Haoyu Yang, Zongyi Li, Kumara Sastry, Saumyadip Mukhopadhyay, Anima Anandkumar, Brucek Khailany, Vivek Singh, and Haoxing Ren. 2022a. Large Scale Mask Optimization Via Convolutional Fourier Neural Operator and Litho-Guided Self Training. arXiv preprint arXiv:2207.04056 (2022).

[59]

Haoyu Yang, Zongyi Li, Kumara Sastry, Saumyadip Mukhopadhyay, Mark Kilgard, Anima Anandkumar, Brucek Khailany, Vivek Singh, and Haoxing Ren. 2022b. Generic lithography modeling with dual-band optics-inspired neural networks. In Proceedings of the 59th ACM/IEEE Design Automation Conference. 973–978.

Digital Library

[60]

Wei Ye, Mohamed Baker Alawieh, Yibo Lin, and David Z Pan. 2019. LithoGAN: End-to-end lithography modeling with generative adversarial networks. In Proceedings of the 56th Annual Design Automation Conference 2019. 1–6.

Digital Library

[61]

Shangting You, Jiaao Guan, Jeffrey Alido, Henry H Hwang, Ronald Yu, Leilani Kwe, Hao Su, and Shaochen Chen. 2020. Mitigating scattering effects in light-based three-dimensional printing using machine learning. Journal of Manufacturing Science and Engineering 142, 8 (2020), 081002.

[62]

Guangyuan Zhao, Cheng Zheng, Cuifang Kuang, Renjie Zhou, Mohammad M Kabir, Kimani C Toussaint Jr, Wensheng Wang, Liang Xu, Haifeng Li, Peng Xiu, 2018. Nonlinear focal modulation microscopy. Physical review letters 120, 19 (2018), 193901.

Cited By

Yang XSouza MWang KChakravarthula PFu QHeidrich W(2024)End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave ModelSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687640(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687640

Index Terms

Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Two-Photon Neural Lithography Simulator
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Optical and EUV projection lithography

Display Omitted Reviews modeling approaches for DUV and EUV projection lithography.Explains both optical and photoresist models.Demonstrates applications of computational lithography.Provides extensive references to related work in this field. This ...
Fabrication of submicron photon sieve using E-beam lithography and X-ray lithography

In this paper, a new hybrid method to fabricate submicron photon sieve is proposed, where the E-beam lithography and the X-ray lithography are used. It is found that 2.8@mm thickness of the polyimide film, 400nm thickness of the ZEP-520 and 280@mC/cm^2 ...
3D Patterning by means of nanoimprinting, X-ray and two-photon lithography
Proceedings of the 29th international conference on micro and nano engineering

Two lithographic techniques suitable for fabricating complex 3D structures with high spatial resolution are presented and discussed. The first one is based on the combined use of nanoimprint and X-ray lithography. Its technological potential has been ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '23: SIGGRAPH Asia 2023 Conference Papers

December 2023

1113 pages

ISBN:9798400703157

DOI:10.1145/3610548

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 December 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NIH (National Institutes of Health)

Conference

SA '23

Sponsor:

SIGGRAPH

SA '23: SIGGRAPH Asia 2023

December 12 - 15, 2023

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
801
Total Downloads

Downloads (Last 12 months)801
Downloads (Last 6 weeks)136

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang XSouza MWang KChakravarthula PFu QHeidrich W(2024)End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave ModelSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687640(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687640

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents