research-article

Marble: collaborative scheduling of batteryless sensors with meta reinforcement learning

Authors:

Francesco Fraternali,

Bharathan Balaji,

Yuvraj Agarwal,

Rajesh K. GuptaAuthors Info & Claims

BuildSys '21: Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

Pages 140 - 149

https://doi.org/10.1145/3486611.3486670

Published: 17 November 2021 Publication History

Abstract

Batteryless energy-harvesting sensing systems are attractive for low maintenance but face challenges in real-world applications due to low quality of service from sporadic and unpredictable energy availability. To overcome this challenge, recent data-driven energy management techniques optimize energy usage to maximize application performance even in low harvested energy scenarios by learning energy availability patterns in the environment. These techniques require prior knowledge of the environment in which the sensor nodes are deployed to work correctly. In the absence of historical data, the application performance deteriorates.

To overcome this challenge, we describe here the use of meta reinforcement learning to increase the application performance of newly deployed batteryless sensor nodes without historical data. Our system, called Marble, exploits information from other sensor node locations to expedite the learning of newly deployed sensor nodes, and improves application performance in the initial period of deployment. Our evaluation using real-world data traces shows that Marble detects up to 66% more events in low lighting conditions, and up to 25.6% more events on average on the first 3 days of deployment compared to the state-of-the-art.¹

References

[1]

Fayçal Ait Aoudia at al. 2018. RLMan: An Energy Manager Based on Reinforcement Learning for Energy Harvesting Wireless Sensor Networks. IEEE Transactions on Green Communications and Networking (2018).

[2]

Bradford Campbell and Prabal Dutta. 2014. An Energy-Harvesting Sensor Architecture and Toolkit for Building Monitoring and Event Detection. In Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficient Buildings (BuildSys '14). Association for Computing Machinery.

Digital Library

[3]

Roy Chaoming Hsu, Cheng-Ting Liu, and Wei-Ming Lee. 2009. Reinforcement Learning-Based Dynamic Power Management for Energy Harvesting Wireless Sensor Network. In Next-Generation Applied Intelligence, Been-Chian Chien, Tzung-Pei Hong, Shyi-Ming Chen, and Moonis Ali (Eds.).

[4]

Qingping Chi, Hairong Yan, Chuan Zhang, Zhibo Pang, and Li Da Xu. 2014. A Reconfigurable Smart Sensor Interface for Industrial WSN in IoT Environment. IEEE Transactions on Industrial Informatics (2014).

[5]

Man Chu, Hang Li, Xuewen Liao, and Shuguang Cui. 2018. Reinforcement learning-based multiaccess control and battery prediction with energy harvesting in IoT systems. IEEE Internet of Things Journal 6, 2 (2018), 2009--2020.

[6]

Sanio Semiconductor CO. 2007. https://media.digikey.com/pdf/Data%20Sheets/Sanyo%20Energy/Amorphous_Br.pdf.

[7]

Samuel DeBruin, Bradford Campbell, and Prabal Dutta. 2013. Monjolo: An Energy-harvesting Energy Meter Architecture. In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems (Roma, Italy) (SenSys '13). ACM, New York, NY, USA, Article 18, 14 pages.

Digital Library

[8]

Gabriel Martins Dias, Maddalena Nurchis, and Boris Bellalta. 2016. Adapting sampling interval of sensor networks using on-line reinforcement learning. In 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT).

[9]

echelon.com. 2018. https://www.echelon.com/.

[10]

Joshua F Ensworth and Matthew S Reynolds. 2017. Ble-backscatter: ultralow-power IoT nodes compatible with bluetooth 4.0 low energy (BLE) smartphones and tablets. IEEE Transactions on Microwave Theory and Techniques 65, 9 (2017).

[11]

R. C. Hsu et al. 2014. A Reinforcement Learning-Based ToD Provisioning Dynamic Power Management for Sustainable Operation of Energy Harvesting Wireless Sensor Node. IEEE Transactions on Emerging Topics in Computing (2014).

[12]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:1703.03400 [cs.LG]

[13]

Francesco Fraternali, Bharathan Balaji, Yuvraj Agarwal, Luca Benini, and Rajesh K. Gupta. 2018. Pible: Battery-Free Mote for Perpetual Indoor BLE Applications. In Proceedings of the 5th ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building (BuildSys '18). ACM.

[14]

Francesco Fraternali, Bharathan Balaji, Yuvraj Agarwal, and Rajesh K. Gupta. 2020. ACES: Automatic Configuration of Energy Harvesting Sensors with Reinforcement Learning. ACM Trans. Sen. Netw. 16, 4, Article 36 (July 2020), 31 pages.

Digital Library

[15]

Francesco Fraternali, Bharathan Balaji, and Rajesh Gupta. 2018. Scaling Configuration of Energy Harvesting Sensors with Reinforcement Learning. In Proceedings of the 6th International Workshop on Energy Harvesting & Energy-Neutral Sensing Systems (ENSsys '18).

Digital Library

[16]

Francesco Fraternali, Bharathan Balaji, Dhiman Sengupta, Dezhi Hong, and Rajesh K. Gupta. 2020. Ember: Energy Management of Batteryless Event Detection Sensors with Deep Reinforcement Learning (SenSys '20). Association for Computing Machinery, New York, NY, USA.

[17]

Josiah Hester, Lanny Sitanayah, and Jacob Sorber. 2015. Tragedy of the Coulombs: Federating Energy Storage for Tiny, Intermittently-Powered Sensors (SenSys '15).

[18]

Josiah Hester and Jacob Sorber. 2017. Flicker: Rapid Prototyping for the Batteryless Internet-of-Things (SenSys '17). Association for Computing Machinery.

Digital Library

[19]

Josiah Hester, Kevin Storer, and Jacob Sorber. 2017. Timely Execution on Intermittently Powered Batteryless Sensors. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems (SenSys '17).

Digital Library

[20]

Jason Hsu, Sadaf Zahedi, Aman Kansal, Mani Srivastava, and Vijay Raghunathan. 2006. Adaptive Duty Cycling for Energy Harvesting Systems. In ISLPED'06 Proceedings of the 2006 Int Symposium on Low Power Electronics and Design.

[21]

Jui-Yang Hsu, Yuan-Jui Chen, and Hung-yi Lee. 2020. Meta learning for end-to-end low-resource speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7844--7848.

[22]

R. C. Hsu, C. T. Liu, K. C. Wang, and W. M. Lee. 2009. QoS-Aware Power Management for Energy Harvesting Wireless Sensor Network Utilizing Reinforcement Learning. In 2009 International Conference on Computational Science and Engineering, Vol. 2. 537--542.

Digital Library

[23]

Hrishikesh Jayakumar, Kangwoo Lee, Woo Suk Lee, Arnab Raha, Younghyun Kim, and Vijay Raghunathan. 2014. Powering the internet of things. In Proceedings of the 2014 international symposium on Low power electronics and design. ACM.

Digital Library

[24]

Aman Kansal, Jason Hsu, Sadaf Zahedi, and Mani B. Srivastava. 2007. Power Management in Energy Harvesting Sensor Networks. ACM Trans. Embed. Comput. Syst. 6, 4, Article 32 (Sept. 2007).

Digital Library

[25]

Bryce Kellogg, Aaron Parks, Shyamnath Gollakota, Joshua R Smith, and David Wetherall. 2014. Wi-Fi backscatter: Internet connectivity for RF-powered devices. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 607--618.

Digital Library

[26]

Azam Khan and Kasper Hornbæk. 2011. Big Data from the Built Environment. In Proceedings of the 2nd International Workshop on Research in the Large. Association for Computing Machinery.

Digital Library

[27]

Victor Lawson and Lakshmish Ramaswamy. 2015. Data Quality and Energy Management Tradeoffs in Sensor Service Clouds. In Big Data (Big Data Congress), 2015 IEEE International Congress on. IEEE, 749--752.

[28]

Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2018. RLlib: Abstractions for Distributed Reinforcement Learning. arXiv:1712.09381 [cs.AI]

[29]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).

[30]

Brandon Lucia, Vignesh Balaji, Alexei Colin, Kiwan Maeng, and Emily Ruppel. 2017. Intermittent computing: Challenges and opportunities. In 2nd Summit on Advances in Programming Languages (SNAPL 2017). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

[31]

Yubo Luo et al. 2019. Spoton: Just-in-time active event detection on energy autonomous sensing systems. Brief Presentations Proceedings (RTAS 2019) (2019).

[32]

Kiwan Maeng, Alexei Colin, and Brandon Lucia. 2017. Alpaca: Intermittent execution without checkpoints. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1--30.

Digital Library

[33]

Paul Martin, Zainul Charbiwala, and Mani Srivastava. 2012. DoubleDip: Leveraging Thermoelectric Harvesting for Low Power Monitoring of Sporadic Water Use. In 10th ACM Conference on Embedded Network Sensor Systems (SenSys '12).

Digital Library

[34]

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2017. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141 (2017).

[35]

C. Moser, L. Thiele, D. Brunelli, and L. Benini. 2010. Adaptive Power Management for Environmentally Powered Systems. IEEE Trans. Comput. (2010).

[36]

Abdulmajid Murad, Kerstin Bach, Frank Alexander Kraemer, and Gavin Taylor. 2019. IoT Sensor Gym: Training Autonomous IoT Devices with Deep Reinforcement Learning. In Proceedings of the 9th International Conference on the Internet of Things. Association for Computing Machinery.

Digital Library

[37]

Abdulmajid Murad, Frank Alexander Kraemer, Kerstin Bach, and Gavin Taylor. 2019. Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning. arXiv preprint arXiv:1905.04181 (2019).

[38]

Abdulmajid Murad, Frank Alexander Kraemer, Kerstin Bach, and Gavin Taylor. 2020. Information-Driven Adaptive Sensing Based on Deep Reinforcement Learning. In 10th International Conference on the Internet of Things (IoT '20).

[39]

Saman Naderiparizi and et al. 2015. Self-localizing Battery-free Cameras. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15).

[40]

Saman Naderiparizi, Aaron N Parks, Zerina Kapetanovic, Benjamin Ransford, and Joshua R Smith. 2015. WISPCam: A battery-free RFID camera. In RFID (RFID), 2015 IEEE International Conference on. IEEE, 166--173.

[41]

Matteo Nardello, Harsh Desai, Davide Brunelli, and Brandon Lucia. 2019. Camaroptera: A Batteryless Long-Range Remote Visual Sensing System (ENSsys'19).

Digital Library

[42]

Panasonic. 2018. https://media.digikey.com/pdf/Data%20Sheets/Panasonic%20Electronic%20Components/SG_Series_LowTemp.pdf.

[43]

products.currentbyge.com. 2017. https://products.currentbyge.com/sites/products.currentbyge.com/files/documents/document_file/DT106-GE-Wireless-Occupancy-Sensor-Wall-Mount-Installation-Guide.pdf.

[44]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. (2017).

[45]

Shaswot Shresthamali and et al. 2017. Adaptive Power Management in Solar Energy Harvesting Sensor Node Using Reinforcement Learning. ACM Transactions on Embedded Computing Systems (TECS) (2017).

[46]

Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 (2017).

[47]

Sujesha Sudevalayam and Purushottam Kulkarni. 2011. Energy harvesting sensor nodes: Survey and implications. IEEE Communications Surveys & Tutorials (2011).

[48]

Richard S Sutton, Andrew G Barto, Francis Bach, et al. 1998. Reinforcement learning: An introduction. MIT press.

[49]

Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher B Kendall, Michael B Gotway, and Jianming Liang. 2016. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging 35, 5 (2016), 1299--1312.

[50]

Vamsi Talla, Bryce Kellogg, Shyamnath Gollakota, and Joshua R Smith. 2017. Battery-free cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 25.

Digital Library

[51]

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. 2019. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096 (2019).

[52]

Adrian Udenze and Klaus McDonald-Maier. 2009. Direct reinforcement learning for autonomous power configuration and control in wireless networks. In Adaptive Hardware and Systems, 2009. AHS 2009. NASA/ESA Conference on. IEEE.

Digital Library

[53]

Sennur Ulukus, Aylin Yener, Elza Erkip, Osvaldo Simeone, Michele Zorzi, Pulkit Grover, and Kaibin Huang. 2015. Energy harvesting wireless communications: A review of recent advances. IEEE Journal on Selected Areas in Communicat (2015).

[54]

utc.com. 2018. https://www.utc.com/.

[55]

Christopher M Vigorito, Deepak Ganesan, and Andrew G Barto. 2007. Adaptive control of duty cycling in energy-harvesting wireless sensor networks. In 2007 4th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks. IEEE, 21--30.

[56]

Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3--4 (1992), 279--292.

[57]

Thomas Weng, Bharathan Balaji, Seemanta Dutta, Rajesh Gupta, and Yuvraj Agarwal. 2011. Managing plug-loads for demand response within buildings. In Proceedings of the Third ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Buildings. 13--18.

Digital Library

[58]

Kok-Lim Alvin Yau and et al. 2012. Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues. Journal of Network and Computer Applications (2012).

Cited By

Giuliano FPagano ACroce DVitale GTinnirello I(2023)Adaptive Algorithms for Batteryless LoRa-Based SensorsSensors10.3390/s2314656823:14(6568)Online publication date: 21-Jul-2023
https://doi.org/10.3390/s23146568
Boovaraghavan SChen CMaravi ACzapik MZhang YHarrison CAgarwal Y(2023)MitesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808657:1(1-32)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580865

Index Terms

Marble: collaborative scheduling of batteryless sensors with meta reinforcement learning
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Sensor networks
2. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
      1. Multi-agent systems

Recommendations

Marble: collaborative scheduling of batteryless sensors with meta reinforcement learning
BuildSys '21: Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

Batteryless energy-harvesting sensing systems are attractive for low maintenance but face challenges in real-world applications due to the low quality of service from sporadic and unpredictable energy availability. To overcome this challenge, recent ...
ACES: Automatic Configuration of Energy Harvesting Sensors with Reinforcement Learning

Many modern smart building applications are supported by wireless sensors to sense physical parameters, given the flexibility they offer and the reduced cost of deployment. However, most wireless sensors are powered by batteries today, and large ...
Pible: battery-free mote for perpetual indoor BLE applications
BuildSys '18: Proceedings of the 5th Conference on Systems for Built Environments

Smart building applications require a large-scale deployment of sensors distributed across the environment. Recent innovations in smart environments are driven by wireless networked sensors as they are easy to deploy. However, replacing these batteries ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

BuildSys '21: Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

November 2021

388 pages

ISBN:9781450391146

DOI:10.1145/3486611

General Chair:
Xiaofan (Fred) Jiang
Columbia University
,
Program Chairs:
Omprakash Gnawali
University of Houston
,
Zoltan Nagy
The University of Texas Austin

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEnergy: ACM Special Interest Group on Energy Systems and Informatics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CONIX Research Center

Conference

BuildSys '21

Sponsor:

SIGEnergy

BuildSys '21: The 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

November 17 - 18, 2021

Coimbra, Portugal

Acceptance Rates

BuildSys '21 Paper Acceptance Rate 28 of 107 submissions, 26%;

Overall Acceptance Rate 148 of 500 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)4

Reflects downloads up to 09 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Giuliano FPagano ACroce DVitale GTinnirello I(2023)Adaptive Algorithms for Batteryless LoRa-Based SensorsSensors10.3390/s2314656823:14(6568)Online publication date: 21-Jul-2023
https://doi.org/10.3390/s23146568
Boovaraghavan SChen CMaravi ACzapik MZhang YHarrison CAgarwal Y(2023)MitesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808657:1(1-32)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580865

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten