demonstration

Demo: On-Device Video Analysis with LLMs

Authors:

Chao ZhangAuthors Info & Claims

HOTMOBILE '24: Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications

Page 153

https://doi.org/10.1145/3638550.3643052

Published: 28 February 2024 Publication History

Get Access

Abstract

We present a new on-device pipeline that efficiently summarizes lecture videos and provides relevant answers directly from a smartphone. We utilize widely accessible tools like OCR and Vosk speech-to-text, coupled with powerful large language models (LLMs), to identify crucial sentences and generate summaries. By harnessing the capabilities of LLMs and the computational power of mobile devices, we fine-tune and quantize BERT and GPT-2 to achieve efficient lecture video summarization and question answering on consumer-grade smartphones like the Pixel 8 Pro. Notably, this approach eliminates the need for cloud APIs, ensuring enhanced user privacy and minimal mobile data usage.

https://www.youtube.com/shorts/zwGdONlKays

References

[1]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018).

Google Scholar

[2]

Haopeng Li, Qiuhong Ke, Mingming Gong, and Tom Drummond. 2023. Progressive Video Summarization via Multimodal Self-supervised Learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5584--5593.

Crossref

Google Scholar

[3]

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019).

Google Scholar

Cited By

View all

Piccialli FChiaro DQi PBellandi VDamiani E(2025)Federated and edge learning for large language modelsInformation Fusion10.1016/j.inffus.2024.102840117(102840)Online publication date: May-2025
https://doi.org/10.1016/j.inffus.2024.102840
Zhang SMa YFang LJia HD'Alfonso SKostakos VKostakos VKay JHoang T(2024)Enabling On-Device LLMs Personalization with Smartphone SensingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677545(186-190)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677545

Index Terms

Demo: On-Device Video Analysis with LLMs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Video summarization

Recommendations

DEMO: Starving Permission-Hungry Android Apps Using SecuRank
CCS '16: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security

We demonstrate SecuRank, a tool that can be employed by Android smartphone users to replace their currently installed apps with functionally-similar ones that require less sensitive access to their device. SecuRank works by using text mining on the app ...
Demo: Reptor: Enabling API Virtualization on Android for Platform Openness
MobiSys '17: Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services

We demonstrate Reptor, a bytecode instrumentation tool enabling API virtualization on Android. It provides a general way to alter functionality of platform APIs on Android. With Reptor, third-party developers can modify the behavior of platform APIs ...
Enabling On-Device LLMs Personalization with Smartphone Sensing
UbiComp '24: Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing

This demo presents a novel end-to-end framework that combines on-device large language models (LLMs) with smartphone sensing technologies to achieve context-aware and personalized services. The framework addresses critical limitations of current ...

Comments

Information & Contributors

Information

Published In

HOTMOBILE '24: Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications

February 2024

167 pages

ISBN:9798400704970

DOI:10.1145/3638550

Chair:
Nigel Davies,
Program Chair:
Chenren Xu

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 February 2024

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

HOTMOBILE '24

Sponsor:

SIGMOBILE

HOTMOBILE '24: 25th International Workshop on Mobile Computing Systems and Applications

February 28 - 29, 2024

CA, San Diego, USA

Acceptance Rates

Overall Acceptance Rate 96 of 345 submissions, 28%

Upcoming Conference

HOTMOBILE '25

Sponsor:
sigmobile

The 26th International Workshop on Mobile Computing Systems and Applications

February 26 - 27, 2025

La Quinta , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
84
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)5

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Piccialli FChiaro DQi PBellandi VDamiani E(2025)Federated and edge learning for large language modelsInformation Fusion10.1016/j.inffus.2024.102840117(102840)Online publication date: May-2025
https://doi.org/10.1016/j.inffus.2024.102840
Zhang SMa YFang LJia HD'Alfonso SKostakos VKostakos VKay JHoang T(2024)Enabling On-Device LLMs Personalization with Smartphone SensingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677545(186-190)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677545

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

DEMO: Starving Permission-Hungry Android Apps Using SecuRank

Demo: Reptor: Enabling API Virtualization on Android for Platform Openness

Enabling On-Device LLMs Personalization with Smartphone Sensing