Color event enhanced single-exposure HDR imaging
Article No.: 156, Pages 1399 - 1407
Abstract
Single-exposure high dynamic range (HDR) imaging aims to reconstruct the wide-range intensities of a scene by using its single low dynamic range (LDR) image, thus providing significant efficiency. Existing methods pay high attention to restoring the luminance by inversing the tone-mapping process, while the color in the over-/under-exposed area cannot be well restored due to the information loss of the single LDR image. To address this issue, we introduce color events into the imaging pipeline, which record asynchronous pixel-wise color changes in a high dynamic range, enabling edge-like scene perception under challenging lighting conditions. Specifically, we propose a joint framework that incorporates color events and a single LDR image to restore both content and color of an HDR image, where an exposure-aware transformer (EaT) module is designed to propagate the informative hints, provided by the normal-exposed LDR regions and the event streams, to the missing areas. In this module, an exposure-aware mask is estimated to suppress distractive information and strengthen the restoration of the over-/under-exposed regions. To our knowledge, we are the first to use color events to enhance single-exposure HDR imaging. We also contribute corresponding datasets, consisting of synthesized datasets and a real-world dataset collected by a DAVIS346-color camera. The datasets can be found at https://www.kaggle.com/datasets/mengyaocui/ce-hdr. Extensive experiments demonstrate the effectiveness of the proposed method.
References
[1]
Akyüz, A. O.; Fleming, R.; Riecke, B. E.; Reinhard, E.; and Bülthoff, H. H. 2007. Do HDR displays support LDR content? A psychophysical evaluation. ACM Trans. Graph., 26(3): 38-es.
[2]
Bardow, P.; Davison, A. J.; and Leutenegger, S. 2016. Simultaneous Optical Flow and Intensity Estimation From an Event Camera. In Proc. CVPR, 884-892.
[3]
Chen, X.; Liu, Y.; Zhang, Z.; Qiao, Y.; and Dong, C. 2021a. Hdrunet: Single image hdr reconstruction with denoising and dequantization. In Proc. CVPR, 354-363.
[4]
Chen, Z.; Zheng, Q.; Niu, P.; Tang, H.; and Pan, G. 2021b. Indoor Lighting Estimation Using an Event Camera. In Proc. CVPR, 14760-14770.
[5]
Choi, S.; Cho, J.; Song, W.; Choe, J.; Yoo, J.; and Sohn, K. 2020. Pyramid Inter-Attention for High Dynamic Range Imaging. Sensors, 20(18): 5102.
[6]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
[7]
Eilertsen, G.; Kronander, J.; Denes, G.; Mantiuk, R. K.; and Unger, J. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM Trans. Graph., 36(6): 1-15.
[8]
Endo, Y.; Kanamori, Y.; and Mitani, J. 2017. Deep reverse tone mapping. ACM Trans. Graph., 36(6): 1-10.
[9]
Guo, X.; Yang, H.; and Huang, D. 2021. Image inpainting via conditional texture and structure dual generation. In Proc. ICCV, 14134-14143.
[10]
Han; and et al. 2020. Neuromorphic camera guided high dynamic range imaging. In Proc. CVPR, 1730-1739.
[11]
Huo, Y.; Yang, F.; Dong, L.; and Brost, V. 2014. Physiological inverse tone mapping based on retina response. The Visual Computer, 30: 507-517.
[12]
Kim, J.; Lee, S.; and Kang, S.-J. 2021. End-to-end differentiable learning to hdr image synthesis for multi-exposure images. In Proc. AAAI, volume 35, 1780-1788.
[13]
Kovaleski, R. P.; and Oliveira, M. M. 2014. High-quality reverse tone mapping for a wide range of exposures. In SIBGRAPI Conference on Graphics, Patterns and Images, 49-56.
[14]
Li, H.; and Wu, X. 2019. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Trans. Image Process., 28(5): 2614-2623.
[15]
Liu, Y.; Gutierrez-Barragan, F.; Ingle, A.; Gupta, M.; and Velten, A. 2022. Single-photon camera guided extreme dynamic range imaging. In Proc. WACV, 1575-1585.
[16]
Liu, Y.-L.; Lai, W.-S.; Chen, Y.-S.; Kao, Y.-L.; Yang, M.-H.; Chuang, Y.-Y.; and Huang, J.-B. 2020. Single-image HDR reconstruction by learning to reverse the camera pipeline. In Proc. CVPR, 1651-1660.
[17]
Ma, J.; Liang, P.; Yu, W.; Chen, C.; Guo, X.; Wu, J.; and Jiang, J. 2020. Infrared and visible image fusion via detail preserving adversarial learning. Inf. Fusion, 54: 85-98.
[18]
Mantiuk, R.; Kim, K. J.; Rempel, A. G.; and Heidrich, W. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph., 30(4): 1-14.
[19]
Marnerides, D.; Bashford-Rogers, T.; Hatchett, J.; and Debattista, K. 2018. Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. In Comput. Graph. Forum., volume 37, 37-49.
[20]
Masia, B.; Serrano, A.; and Gutierrez, D. 2017. Dynamic range expansion based on image statistics. Multimedia Tools and Applications, 76: 631-648.
[21]
Metzler, C. A.; Ikoma, H.; Peng, Y.; and Wetzstein, G. 2020. Deep Optics for Single-Shot High-Dynamic-Range Imaging. In Proc. CVPR, 1372-1382.
[22]
Nazeri, K.; Ng, E.; Joseph, T.; Qureshi, F. Z.; and Ebrahimi, M. 2019. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212.
[23]
Niu, Y.; Wu, J.; Liu, W.; Guo, W.; and Lau, R. W. H. 2021. HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions. IEEE Trans. Image Process., 30: 3885-3896.
[24]
Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252.
[25]
Samra, R.; Mitra, K.; and Shedligeri, P. 2023. High-Speed HDR Video Reconstruction from Hybrid Intensity Frames and Events. In Proc. CVMI, 179-190. Springer.
[26]
Scheerlinck, C.; Rebecq, H.; Stoffregen, T.; Barnes, N.; Mahony, R.; and Scaramuzza, D. 2019. CED: Color event camera dataset. In Proc. CVPRW, 1684-1693.
[27]
Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; and Woo, W.-c. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28: 802-810.
[28]
Taverni, G.; Moeys, D. P.; Li, C.; Cavaco, C.; Motsnyi, V.; Bello, D. S. S.; and Delbruck, T. 2018. Front and back illuminated dynamic and active pixel vision sensors comparison. IEEE Trans. Circuits and Systems II: Express Briefs, 65(5): 677-681.
[29]
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30.
[30]
Wang, B.; He, J.; Yu, L.; Xia, G.; and Yang, W. 2020. Event Enhanced High-Quality Image Recovery. In Proc. ECCV, volume 12358, 155-171.
[31]
Wang, H.; Ye, M.; Zhu, X.; Li, S.; Zhu, C.; and Li, X. 2022. KUNet: Imaging Knowledge-Inspired Single HDR Image Reconstruction. In IJCAI-ECAI.
[32]
Wang, L.; and Yoon, K. 2022. Deep Learning for HDR Imaging: State-of-the-Art and Future Trends. IEEE Trans. Pattern Anal. Mach. Intell., 44(12): 8874-8895.
[33]
Woo, S.; Park, J.; Lee, J.-Y.; and Kweon, I. S. 2018. Cbam: Convolutional block attention module. In Proc. ECCV, 3-19.
[34]
Wu, S.; Xu, J.; Tai, Y.-W.; and Tang, C.-K. 2018. Deep high dynamic range imaging with large foreground motions. In Proc. ECCV, 117-132.
[35]
Yang, Y.; Han, J.; Liang, J.; Sato, I.; and Shi, B. 2023. Learning event guided high dynamic range video reconstruction. In Proc. CVPR, 13924-13934.
[36]
Zeng, H.; Cai, J.; Li, L.; Cao, Z.; and Zhang, L. 2022. Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time. IEEE Trans. Pattern Anal. Mach. Intell., 44(4): 2058-2073.
[37]
Zhang, R.; Zhu, J.-Y.; Isola, P.; Geng, X.; Lin, A. S.; Yu, T.; and Efros, A. A. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph., 36(4): 1-11.
Index Terms
- Color event enhanced single-exposure HDR imaging
Index terms have been assigned to the content through auto-classification.
Recommendations
Deep Snapshot HDR Imaging Using Multi-exposure Color Filter Array
Computer Vision – ACCV 2020AbstractIn this paper, we propose a deep snapshot high dynamic range (HDR) imaging framework that can effectively reconstruct an HDR image from the RAW data captured using a multi-exposure color filter array (ME-CFA), which consists of a mosaic pattern of ...
Color correction of high dynamic range images at HDR-level
SIGGRAPH '07: ACM SIGGRAPH 2007 postersIn recent research in representing realistic appearance of materials, HDR(High Dynamic Range) images constructed from LDR(Low Dynamic Range) images taken with different exposure time using a digital camera are used to measure radiance of materials like ...
Evaluating the Color Fidelity of ITMOs and HDR Color Appearance Models
Special Issue SAP 2015With the increasing availability of high-dynamic-range (HDR) displays comes the need to remaster existing content in a way that takes advantage of the extended range of luminance and contrast that such displays offer. At the same time, it is crucial ...
Comments
Information & Contributors
Information
Published In
February 2024
23861 pages
ISBN:978-1-57735-887-9
Copyright © 2024 Association for the Advancement of Artificial Intelligence.
Sponsors
- Association for the Advancement of Artificial Intelligence
Publisher
AAAI Press
Publication History
Published: 07 January 2025
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Jan 2025