User Interface Softbots

Book One

Volume

Proceedings of the AAAI Conference on Artificial Intelligence, 17

Intelligent Systems Demos

An interface softbot, or ibot, controls an interactive system through the graphical user interface, as human users do, without relying on an application programming interface (API) or access to source code. Our work has produced a programmable substrate for ibots, containing sensors, effectors, and skeleton controllers. Sensor modules take pixel-level input from the display, run the data through image processing algorithms, and build a representation of visible interface objects. Effector modules generate mouse and keyboard gestures to manipulate these objects. These sensors and effectors act as eyes and hands to be managed by a controller appropriate for an application domain. This demonstration will show ibots interacting with several unmodified, off-the-shelf software applications, carrying out tasks that we might find useful in everyday computer use.