skip to main content
research-article
Open access

Interaction Harvesting: A Design Probe of User-Powered Widgets

Published: 27 September 2023 Publication History

Abstract

Whenever a user interacts with a device, mechanical work is performed to actuate the user interface elements; the resulting energy is typically wasted, dissipated as sound and heat. Previous work has shown that many devices can be powered entirely from this otherwise wasted user interface energy. For these devices, wires and batteries, along with the related hassles of replacement and charging, become unnecessary and onerous. So far, these works have been restricted to proof-of-concept demonstrations; a specific bespoke harvesting and sensing circuit is constructed for the application at hand. The challenge of harvesting energy while simultaneously sensing fine-grained input signals from diverse modalities makes prototyping new devices difficult. To fill this gap, we present a hardware toolkit which provides a common electrical interface for harvesting energy from user interface elements. This facilitates exploring the composability, utility, and breadth of enabled applications of interaction-powered smart devices. We design a set of "energy as input" harvesting circuits, a standard connective interface with 3D printed enclosures, and software libraries to enable the exploration of devices where the user action generates the energy needed to perform the device's primary function. This exploration culminated in a demonstration campaign where we prototype several exemplar popular toys and gadgets, including battery-free Bop-It--- a popular 90s rhythm game, an electronic Etch-a-sketch, a "Simon-Says"-style memory game, and a service rating device. We run exploratory user studies to understand how generativity, creativity, and composability are hampered or facilitated by these devices. These demonstrations, user study takeaways, and the toolkit itself provide a foundation for building interactive and user-focused gadgets whose usability is not affected by battery charge and whose service lifetime is not limited by battery wear.

References

[1]
Mikhail Afanasov, Naveed Bhatti, Dennis Campagna, Giacomo Caslini, Fabio Massimo Centonze, Koustabh Dolui, Andrea Maioli, Erica Barone, Muhammad Hamad Alizai, Junaid Haroon Siddiqui, and Luca Mottola. 2020. Battery-less Zero-maintenance Embedded Sensing at the Mithraeum of Circus Maximus. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (SenSys '20). Association for Computing Machinery, New York, NY, USA, 1--14.
[2]
ZF Friedrichshafen AG. 2023. AFIG-0007. https://switches-sensors.zf.com/us/eh-part/afig-0007/
[3]
Nivedita Arora and Gregory D Abowd. 2018. ZEUSSS: Zero energy ubiquitous sound sensing surface leveraging triboelectric nanogenerator and analog backscatter communication. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings. 81--83.
[4]
Nivedita Arora, Thad Starner, and Gregory D Abowd. 2020. SATURN: An introduction to the Internet of Materials. Commun. ACM 63, 12 (2020), 92--99.
[5]
Nivedita Arora, Steven L Zhang, Fereshteh Shahmiri, Diego Osorio, Yi-Cheng Wang, Mohit Gupta, Zhengjun Wang, Thad Starner, Zhong Lin Wang, and Gregory D Abowd. 2018. SATURN: A thin and flexible self-powered microphone leveraging triboelectric nanogenerator. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 1--28.
[6]
Akash Badshah, Sidhant Gupta, Gabe Cohn, Nicolas Villar, Steve Hodges, and Shwetak N. Patel. 2011. Interactive Generator: A Self-Powered Haptic Feedback Device. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). Association for Computing Machinery, New York, NY, USA, 2051--2054. https://doi.org/10.1145/1978942.1979240
[7]
Abu Bakar, Rishabh Goel, Jasper de Winkel, Jason Huang, Saad Ahmed, Bashima Islam, Przemysław Pawełczak, Kasım Sinan Yıldırım, and Josiah Hester. 2022. Protean: An Energy-Efficient and Heterogeneous Platform for Adaptive and Hardware-Accelerated Battery-Free Computing. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems (SenSys '22). 207--221.
[8]
Tim Campbell, Eric Larson, Gabe Cohn, Jon Froehlich, Ramses Alcaide, and Shwetak N Patel. 2010. WATTR: a method for self-powered wireless sensing of water activity in the home. In Proceedings of the 12th ACM international conference on Ubiquitous computing. 169--172.
[9]
Matthew J Chabalko, Mohsen Shahmohammadi, and Alanson P Sample. 2017. Quasistatic cavity resonance for ubiquitous wireless power transfer. PloS one 12, 2 (2017), e0169045.
[10]
Pololu Corporation. 2023. Micro Metal Gearmotors. https://www.pololu.com/category/60/micro-metal-gearmotors
[11]
Joanna Dauner, Matthew Glisson, and Mustafa Karagozler. 2014. Electric flora: an interactive energy harvesting installation. In Proceedings of the 2014 companion publication on Designing interactive systems. 125--128.
[12]
Jasper de Winkel, Vito Kortbeek, Josiah Hester, and Przemysław Pawełczak. 2020. Battery-Free Game Boy. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3, Article 111 (Sept. 2020), 34 pages. https://doi.org/10.1145/3411839
[13]
Samuel DeBruin, Branden Ghena, Ye-Sheng Kuo, and Prabal Dutta. 2015. Powerblade: A low-profile, true-power, plug-through energy meter. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. 17--29.
[14]
Bradley Denby and Brandon Lucia. 2020. Orbital edge computing: Nanosatellite constellations as a new class of computer system. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 939--954.
[15]
James Devine, Joe Finney, Peli de Halleux, Michał Moskal, Thomas Ball, and Steve Hodges. 2018. MakeCode and CODAL: intuitive and efficient embedded systems programming for education. ACM SIGPLAN Notices 53, 6 (2018), 19--30.
[16]
Christine Dierk, Molly Jane Pearce Nicholas, and Eric Paulos. 2018. AlterWear: battery-free wearable displays for opportunistic interactions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--11.
[17]
EnOcean. 2023. EnOcean Self Powered Sensors and Kinetic Switches. https://www.enocean.com. Last accessed: Feb. 6, 2023.
[18]
Chuhan Gao, Yilong Li, and Xinyu Zhang. 2018. LiveTag: Sensing human-object interaction through passive chipless WiFi tags. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 533--546.
[19]
Saul Greenberg and Chester Fitchett. 2001. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th annual ACM symposium on User interface software and technology. 209--218.
[20]
Manoj Gulati, Farshid Salemi Parizi, Eric Whitmire, Sidhant Gupta, Shobha Sundar Ram, Amarjeet Singh, and Shwetak N Patel. 2018. CapHarvester: A stick-on capacitive energy harvester using stray electric field from AC power lines. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--20.
[21]
Josiah Hester, Travis Peters, Tianlong Yun, Ronald Peterson, Joseph Skinner, Bhargav Golla, Kevin Storer, Steven Hearndon, Kevin Freeman, Sarah Lord, Ryan Halter, David Kotz, and Jacob Sorber. 2016. Amulet: An Energy-Efficient, Multi-Application Wearable Platform. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems (SenSys '16). ACM, New York, NY, USA, 216--229. https://doi.org/10.1145/2994551.2994554
[22]
Josiah Hester and Jacob Sorber. 2017. Flicker: Rapid prototyping for the batteryless internet-of-things. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems (SenSys '17). 1--13.
[23]
Meng-Ju Hsieh, Jr-Ling Guo, Chin-Yuan Lu, Han-Wei Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2019. RFTouchPads: batteryless and wireless modular touch sensor pads based on RFID. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 999--1011.
[24]
Meng-Ju Hsieh, Rong-Hao Liang, Da-Yuan Huang, Jheng-You Ke, and Bing-Yu Chen. 2018. RFIBricks: Interactive building blocks based on RFID. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1--10.
[25]
Vikram Iyer, Elyas Bayati, Rajalakshmi Nandakumar, Arka Majumdar, and Shyamnath Gollakota. 2018. Charging a smartphone across a room using lasers. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--21.
[26]
Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, and Jason Hong. 2018. Wish: Towards a wireless shape-aware world using passive rfids. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 428--441.
[27]
Mustafa Emre Karagozler, Ivan Poupyrev, Gary K. Fedder, and Yuri Suzuki. 2013. Paper Generators: Harvesting Energy from Touching, Rubbing and Sliding. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 23--30. https://doi.org/10.1145/2501988.2502054
[28]
Majeed Kazemitabaar, Jason McPeak, Alexander Jiao, Liang He, Thomas Outing, and Jon E Froehlich. 2017. Makerwear: A tangible approach to interactive wearable creation for children. In Proceedings of the 2017 chi conference on human factors in computing systems. 133--145.
[29]
Christopher Kraemer, Amy Guo, Saad Ahmed, and Josiah Hester. 2022. Battery-free MakeCode: Accessible Programming for Intermittent Computing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1--35.
[30]
David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--17.
[31]
Hanchuan Li, Eric Brockmeyer, Elizabeth J Carter, Josh Fromm, Scott E Hudson, Shwetak N Patel, and Alanson Sample. 2016. Paperid: A technique for drawing functional battery-free wireless interfaces on paper. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 5885--5896.
[32]
Long Lin, Yannan Xie, Sihong Wang, Wenzhuo Wu, Simiao Niu, Xiaonan Wen, and Zhong Lin Wang. 2013. Triboelectric active sensor array for self-powered static and dynamic pressure detection and tactile imaging. ACS nano 7, 9 (2013), 8266--8274.
[33]
Vincent Liu, Aaron Parks, Vamsi Talla, Shyamnath Gollakota, David Wetherall, and Joshua R Smith. 2013. Ambient backscatter: Wireless communication out of thin air. ACM SIGCOMM computer communication review 43, 4 (2013), 39--50.
[34]
EcoCentricNow LLC. 2023. Shake Light 40-B Flashlight, NiMH Battery. https://ecocentricnow.com/product/shake-light-40b-flashlight-handheld-led-lights/
[35]
Yogesh Kumar Meena, Krishna Seunarine, Deepak Ranjan Sahoo, Simon Robinson, Jennifer Pearson, Chi Zhang, Matt Carnie, Adam Pockett, Andrew Prescott, Suzanne K Thomas, et al. 2020. PV-Tiles: Towards Closely-Coupled Photovoltaic and Digital Materials for Useful, Beautiful and Sustainable Interactive Surfaces. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--12.
[36]
Joseph A Paradiso and Mark Feldmeier. 2001. A compact, wireless, self-powered pushbutton controller. In International Conference on Ubiquitous Computing. Springer, 299--304.
[37]
Jung Wook Park, Alishan Hassan, Tingyu Cheng, Rosa Arriaga, and Gregory Abowd. 2021. A Simulation and Prototyping Toolkit for Airflow Energy Harvesting in Vehicles. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 588--589.
[38]
James Pierce and Eric Paulos. 2012. Beyond energy monitors: interaction, energy, and emerging energy systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 665--674.
[39]
James Pierce and Eric Paulos. 2012. Designing Everyday Technologies with Human-Power and Interactive Microgeneration. In Proceedings of the Designing Interactive Systems Conference (DIS '12). Association for Computing Machinery, New York, NY, USA, 602--611. https://doi.org/10.1145/2317956.2318047
[40]
Kimiko Ryokai, Peiqi Su, Eungchan Kim, and Bob Rollins. 2014. EnergyBugs: Energy Harvesting Wearables for Children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). Association for Computing Machinery, New York, NY, USA, 1039--1048. https://doi.org/10.1145/2556288.2557225
[41]
Alanson P Sample, Daniel J Yeager, Pauline S Powledge, Alexander V Mamishev, and Joshua R Smith. 2008. Design of an RFID-based battery-free programmable sensing platform. IEEE transactions on instrumentation and measurement 57, 11 (2008), 2608--2615.
[42]
Edward Sazonov, Haodong Li, Darrell Curry, and Pragasen Pillay. 2009. Self-powered sensors for monitoring of highway bridges. IEEE Sensors Journal 9, 11 (2009), 1422--1429.
[43]
Milijana Surbatovich, Limin Jia, and Brandon Lucia. 2021. Automatically Enforcing Fresh and Consistent Inputs in Intermittent Systems. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI 2021). Association for Computing Machinery, New York, NY, USA, 851--866. https://doi.org/10.1145/3453483.3454081
[44]
Vamsi Talla, Bryce Kellogg, Shyamnath Gollakota, and Joshua R Smith. 2017. Battery-free cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1--20.
[45]
Shan-Yuan Teng, KD Wu, Jacqueline Chen, and Pedro Lopes. 2022. Prolonging VR Haptic Experiences by Harvesting Kinetic Energy from the User. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1--18.
[46]
Hoang Truong, Shuo Zhang, Ufuk Muncuk, Phuc Nguyen, Nam Bui, Anh Nguyen, Qin Lv, Kaushik Chowdhury, Thang Dinh, and Tam Vu. 2018. Capband: Battery-free successive capacitance sensing wristband for hand gesture recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 54--67.
[47]
Nicolas Villar and Steve Hodges. 2010. The Peppermill: A Human-Powered User Interface Device. In Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '10). Association for Computing Machinery, New York, NY, USA, 29--32. https://doi.org/10.1145/1709886.1709893
[48]
Anandghan Waghmare, Qiuyue Xue, Dingtian Zhang, Yuhui Zhao, Shivan Mittal, Nivedita Arora, Ceara Byrne, Thad Starner, and Gregory D Abowd. 2020. UbiquiTouch: Self sustaining ubiquitous touch interfaces. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--22.
[49]
Jingxian Wang, Chengfeng Pan, Haojian Jin, Vaibhav Singh, Yash Jain, Jason I Hong, Carmel Majidi, and Swarun Kumar. 2019. RFID Tattoo: A Wireless Platform for Speech Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1--24.
[50]
Peta Wyeth. 2008. How young children learn to program with sensor, action, and logic blocks. The Journal of the learning sciences 17, 4 (2008), 517--550.
[51]
Weiqing Yang, Jun Chen, Guang Zhu, Jin Yang, Peng Bai, Yuanjie Su, Qingsheng Jing, Xia Cao, and Zhong Lin Wang. 2013. Harvesting energy from the natural vibration of human walking. ACS nano 7, 12 (2013), 11317--11324.
[52]
Eren Yildiz, Saad Ahmed, Bashima Islam, Josiah Hester, and Kasim Sinan Yildirim. 2023. Efficient and Safe I/O Operations for Intermittent Systems. In Proceedings of the Eighteenth European Conference on Computer Systems (EuroSys '23). Association for Computing Machinery, New York, NY, USA, 63--78. https://doi.org/10.1145/3552326.3587435
[53]
Chi Zhang, Sidharth Kumar, and Dinesh Bharadia. 2019. Capttery: Scalable Battery-like Room-level Wireless Power. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. 1--13.
[54]
Dingtian Zhang, Jung Wook Park, Yang Zhang, Yuhui Zhao, Yiyang Wang, Yunzhi Li, Tanvi Bhagwat, Wen-Fang Chou, Xiaojia Jia, Bernard Kippelen, et al. 2020. OptoSense: Towards Ubiquitous Self-Powered Ambient Light Sensing Surfaces. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1--27.
[55]
Tengxiang Zhang, Xin Yi, Chun Yu, Yuntao Wang, Nicholas Becker, and Yuanchun Shi. 2017. TouchPower: Interaction-based power transfer for power-as-needed devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--20.
[56]
Yang Zhang, Yasha Iravantchi, Haojian Jin, Swarun Kumar, and Chris Harrison. 2019. Sozu: Self-Powered Radio Tags for Building-Scale Activity Sensing. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 973--985. https://doi.org/10.1145/3332165.3347952

Cited By

View all
  • (2024)NeuroCamTags: Long-Range, Battery-free, Wireless Sensing with Neuromorphic CamerasProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785298:3(1-25)Online publication date: 9-Sep-2024

Index Terms

  1. Interaction Harvesting: A Design Probe of User-Powered Widgets

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 7, Issue 3
    September 2023
    1734 pages
    EISSN:2474-9567
    DOI:10.1145/3626192
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 September 2023
    Published in IMWUT Volume 7, Issue 3

    Check for updates

    Author Tags

    1. Bop-it
    2. Embedded
    3. Energy Harvesting
    4. Interaction Power

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)451
    • Downloads (Last 6 weeks)42
    Reflects downloads up to 06 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)NeuroCamTags: Long-Range, Battery-free, Wireless Sensing with Neuromorphic CamerasProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785298:3(1-25)Online publication date: 9-Sep-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media