A-Coord Input: Coordinating Auxiliary Input Streams for Augmenting Contextual Pen-Based Interactions Khalad Hasan1, Xing-Dong Yang2, Andrea Bunt1, Pourang Irani1 2 Department of Computer Science, Department of Computing Science University of Manitoba, University of Alberta, Winnipeg, Manitoba, Canada Edmonton, Alberta, Canada {khalad, bunt, irani}@cs.umanitoba.ca [email protected] 1 ABSTRACT The human hand can naturally coordinate multiple finger joints, and simultaneously tilt, press and roll a pen to write or draw. For this reason, digital pens are now embedded with auxiliary input sensors to capture these actions. Prior research on auxiliary input channels has mainly investigated them in isolation of one another. In this work, we explore the coordinated use of two auxiliary channels, a class of interaction techniques we refer to as a-coord input. Through two separate experiments, we explore the design space of a-coord input. In the first study we identify if users can successfully coordinate two auxiliary channels. We found a strong degree of coordination between channels. In a second experiment, we evaluate the effectiveness of acoord input in a task with multiple steps, such as multiparameter selection and manipulation. We find that a-coord input facilitates coordination even with a complex, aforethought sequential task. Overall our results indicate that users can control at least two auxiliary input channels in conjunction which can facilitate a number of common tasks can on the pen. Author Keywords Pen-based interaction; pen roll; pen pressure; pen tilt; dualchannel input. ACM Classification Keywords H.5.2 [Information Interfaces And Presentation]: User Interfaces - Interaction styles. clude rapid access to contextual commands [22], finegrained parameter manipulation [15], and improved stimulus-response compatibility [21]. Naturally, prior work has investigated the design space for each of these pen input channels in isolation of one another, or when merged with pen-tip movement [3, 15-17, 21, 22]. Such research has been instrumental in identifying the fundamental properties and limitations of these auxiliary pen input streams [3, 16, 22]. However, a new collection of results is necessary to explore whether users can control such channels simultaneously, beyond our abilities to do so with highly familiar and well-practiced tasks, such as writing and drawing. If such coordination is possible, this would expand the pen’s interactive space. We build on these earlier results and investigate a-coord input, the [coord]ination of at least two different [a]uxiliary channels, such as roll and pressure, or tilt and roll, on the pen (Figure 1). The a-coord input style raises many human performance questions that warrant long-term research. At this early stage we focus on the most basic questions: 1) can users coordinate two auxiliary channels simultaneously?; 2) can multi-channel coordination extend the bandwidth (number of controllable items) that is available with single auxiliary channels?; 3) how does coordination differ between different auxiliary channels?; and, 4) how can acoord be applied to tasks involving continuous manipulation, such as multi-parameter selection and manipulation? INTRODUCTION The digital pen has evolved into a sophisticated input device, with the ability to capture a large range of natural manipulations such as finger roll, pressure and tilt, through auxiliary input channels. Given these capabilities in comparison to the mouse, it is not surprising that some visionaries tout the pen as becoming a highly relied upon device for the next two decades [2]. Prior research has demonstrated the merits in using the pen’s auxiliary channels. These inPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI’12, May 5–10, 2012, Austin, Texas, USA. Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00. Figure 1 – An illustration of (a) contextual 2D menu interaction with a-coord Tilt+Pressure; and (b) multi-parameter selection and manipulation. We restrict this initial investigation to contextual input, i.e., to tasks for which pen tip movement is not required. Based on the primary features of the pen’s auxiliary channels, we designed two experiments to respond to the above listed questions. Our findings show that a-coord input successful- ly extends the control of auxiliary input from 1D to 2D. We observe a high degree of coordination with 2D contextual tasks, with certain a-coord input styles exhibiting more parallelism than others. Our findings also show that we can apply a-coord input to multi-parameter selection and manipulation, a task that involves continuous manipulation. This latter task also has a clearer two-step delineation than the 2D contextual menus, allowing us to test a-coord input in a situation where one channel is designated as the leading channel and must be held steady while the user operates the second channel. We follow these experiments with an illustration of how carefully composing the pen’s auxiliary inputs can provide a diverse set of interactive techniques. Our contributions include: (1) an examination of the coordinated control of the pen’s auxiliary channels, which we term a-coord input; (2) an extension of such input for 2D contextual tasks; (3) evidence of good coordination with some a-coord input styles; (4) a demonstration of a-coord input’s effectiveness for complex tasks, such as multiparameter selection and manipulation; (5) a demonstration of a varied sample of interactive tasks possible with the pen’s auxiliary input channels. RELATED WORK Our research builds on the benefits and limitations of the pen’s auxiliary input channels, which we review first. We then briefly cover work in the area of parallel input control and conclude this section with a presentation of techniques for multi-parameter selection and manipulation, a task to which we apply our designs of a-coord input. Auxiliary Pen Input Channels Numerous studies have explored the benefits and limitations of each of the pen’s auxiliary input channels [3, 13, 6, 1]. Existing findings with pen pressure, tilt-azimuth (angle around the interaction plane), tilt-altitude (angle between pen and plane) and roll serve as a reference for our design of a-coord input. Pen pressure has received considerable attention in recent years. Ramos and Balakrishnan [14-17], as well as Ren et al. [18] demonstrated that pen pressure is suited for numerous tasks, including menu selection and single parameter manipulation. Studies confirm that users can comfortably control no more than 7±1 discrete pressure levels [13, 16], which can further be improved with proper pressure space discretization techniques [18,16]. Additionally, users can control a limited number of pressure levels with simultaneous pen movement, as in PressureMarks [17], or for fine parameter manipulation, such as with Zliding [15]. The pen’s tilt has an azimuth and an altitude component [22, 23]. Tilt Cursor [21] and the TiltMenu [22], two early systems, respectively showed improved stimulus-response compatibility [21] and strong compatibility with command selection and direct manipulation [22]. The TiltMenu worked well for fewer than 8 discrete items, and some orientations were better than others [22]. A recent study revealed a decreasing power relationship between angular width and pointing performance when using the tilt’s altitude for selection [23]. Pen roll was shown to be useful for mode switching, document navigation, or for fluid parameter manipulation [3, 20]. Bi et al. [3] have shown that targets should be at least 10° in width and within a range of ±80°, to roll with reasonable speed and accuracy. Prior results focusing on each channel in isolation serve as a foundation for the work we present here. We investigate how these channels interact when used in a coordinated manner on the pen. With this knowledge, designers can leverage a-coord input to create improved and fluid interactive pen techniques. Parallel Input Control One potential advantage to using a-coord input is the ability to coordinate the channels simultaneously. Users’ ability to operate multiple degrees-of-freedom of input has been explored in a number of other contexts (e.g., [1, 7, 11]). Jacob et al. [7] characterized input devices as either integral or separable based on whether they allowed users to manipulate multiple DOF simultaneously. Their study revealed the importance of matching the perceptual nature of a task to that of the input device. Other work has examined the degree of parallelism exhibited in specific settings, such as a 3D docking task [11] and in bimanual interaction [1]. We use this body of prior work to inform our visual feedback and our methods for assessing coordination. Parameter Selection and Manipulation Techniques To demonstrate that a-coord input can benefit a range of tasks, we consider its use in multi-parameter selection and manipulation. This task is normally carried out in two distinct steps; to first select a parameter, and then to adjust its value. Numerous techniques have been proposed for fluidly merging multi-parameter selection and manipulation. Flowmenu [6] is a stroke-based interface with a radial layout of regions that define various commands. Selecting a feature takes place by stroking across a wedge-shaped menu item. Adjusting the value of a parameter occurs by tracing radially around the FlowMenu. The FaST sliders interface [12] consists of using marking menus with a typical linear slider. Users first apply a mark, in the marking menu, which then triggers a value adjusting slider. The user then moves the slider in the desired position. An informal user study showed that both FaST sliders and FlowMenus effectively support parameter manipulation, but that FaST sliders were easier for participants to learn [12]. In contrast to the above systems, with a-coord input it is possible that experienced users could execute some degree of parallelism, in that they could begin to manipulate the value of a parameter while they are selecting the parameter. PROPERTIES OF AUXILIARY CHANNELS Key to our investigation is a comparative analysis of the various auxiliary channels on the pen. We do not explore all of the possible channels, such as hover [5], or capacitance based multi-touch [19] as we leave these for future work. In our analysis we include Tilt, Roll, and Pressure. We leave Tilt-Altitude for future work, as it shares similar properties with Tilt-Azimuth (which we explore). We first compare the various features of these channels and then describe the design choices for our study of a-coord input. Channel Properties We distinguish each of these channels along five major axes: range of discrete control, bi-directionality, visuomotor mappings, cyclicality and access method, introduced below and summarized in (Table 1). Range of discrete control. Researchers have identified the number of discrete levels a channel can control. For pressure this number is 7±1 [13, 16], for Roll it is ±80°/10° (smallest allowable angle) or 16 levels [3] and for TiltAzimuth, performance degrades before attaining 8 discrete levels [22]. These ranges place an upper bound on what is possible in terms of item selection. Bi-directionality. Most channels provide a reasonably good control of the input space in the forward and backward movements. Pressure is a slight exception. Because of how the sensors operate, pressure affords better control when moving forward and less control returning from higher to lower values [8]. Bi-directionality allows for better control if the user were to overshoot a desired target. Visuo-motor mapping. Visual feedback (i.e., a mapping from motor to display space) is key for operating auxiliary channel input, particularly in the absence of body-based feedback (i.e., Pressure) [16]. Prior work has employed radial controls for Roll and Tilt, but linear for Pressure. Roll and Pressure can also be mapped to a linear or radial control, respectively. On the other hand, mapping tiltazimuth to a linear control would not be a good match with the corresponding biomechanical operation. Cyclicality. Channel control can be either cyclical or noncyclical. For example, Roll affords cyclical control, as the user can return to the starting point (for example, an angle of 0°) in a single stroke without changing movement direction. In contrast, Pressure can only return to its originating value if the pen were to be lifted. Access method. This feature suggests how quickly a channel can access an item. This can happen sequentially, by going through each value, or by leaping through a number of intermediary values and going directly to an item of interest, as observed in [22]. Only Tilt-Azimuth works this efficiently as one can directly tilt the pen (or leap) to the orientation of interest; all the other channels require sequentially traversing through values in their range. Design Considerations Guided by the comparative analysis above, we restricted our implementation of a-coord input to the following constraints and scope. Visual Feedback In our experiments, visual mappings were congruent with motor movement. We map Tilt to a radial layout, but Pres- sure and Roll are mapped to either a radial or linear layout, to provide flexibility in our visual feedback methods. Selection Techniques Selection is necessary to complete the final step of an action. For Pressure, a quick release or dwell have been preferred over selection with the pen’s barrel button [16]. For Tilt, Tian et al. [22] proposed using the altitude of tilt for selection. For Roll, Bi et al. [3] proposed using quick release. Prior results also show that a button press with the non-dominant hand provides good control and efficiency [10, 16]. We use this latter method in our studies, especially since two channels are being controlled at once. Discretizing Raw Sensory Input Raw sensor information does not always provide an ideal mapping of sensor values to interaction [8]. Researchers have proposed discretizing the input for better control. Pressure input has been discretized into distinct levels using linear [15], quadratic [4], a dynamic fisheye-based [8] or a sigmoid [16] discretization function. We used a hysteresis function similar to that found in [16] for pressure. All other channels employed a one-to-one mapping from raw Tilt or Roll motor displacements to visual effecter. Discrete Levels Bi-Directionality Visuo-Motor Mapping(s) Cyclicality Access Method Roll Pressure Tilt Azimuth 16 7±1 <8 Good Weak Good Radial (P) Linear (P) Linear (S) Radial (S) Cyclical Non-cyclical Cyclical Sequential Sequential Leaping Radial Table 1- A summary of key features of the pen’s auxiliary input channels based on the literature. We used these to guide our design choices. P, S refer to primary, secondary. EXPERIMENT 1 – COORDINATING TWO CHANNELS The goal of this experiment was to explore whether acoord input (a) allows users to effectively coordinate auxiliary channels conjunctively, and (b) extends the number of controllable items with auxiliary input. Without a-coord input, the latter goal could be realized by first applying one channel, a selector, and then the same channel again (i.e. Roll+selector+Roll). The selector would indicate movement into the next dimension. Alternatively one could apply one channel, a selector, and then another channel, but this would resemble a-coord input which makes the selector redundant. Furthermore, sequential operation of two channels does not provide the freedom to re-adjust the first channel after it has been 'locked in’ (i.e., to proceed to the second channel). Therefore, we used the first design as a baseline. Participants and Apparatus Ten right-handed participants (2 females) between the ages of 18 and 35 were recruited for this study. Participants had little or no experience with digital pen input. We used a Wacom Intuos4 tablet with an Intuos4 Art Pen. The pen can produce pressure, tilt and roll values with a maximum of 2048 levels of pressure, and 360° of roll and tilt. We displayed visual feedback in full-screen mode on a 22-inch monitor with a resolution of 1680×1050 pixels. only three input channels: pressure, roll and tilt. We acknowledge that our results may not generalize to all combinations of a-coord inputs, but hope to show that at least some combinations provide clear benefits. We used these three channels with the following parameters. Tasks and Procedure Pressure – We applied a hysteresis function similar to that found in [16]. However, we excluded pressure readings that were simply resulting from the weight of the pen as this could confound our results. The range selected was thus between 819 and 2048 pressure units (where 2048 was approximately 1.5N of force). The initial pressure value was mapped to 0° as indicated in Figure 2. We used a 2D discrete-target selection task. All first level items were arranged in a 360° circular layout (Figure 2). Second level items were placed in concentric rings. We chose this mapping as it would allow us to explore a range of a-coord techniques without introducing any confounds related to unintuitive visuo-motor mappings. The size of each target was determined by the number of items in the menu (i.e., fewer items resulted in larger targets). A target was highlighted in red. The user’s cursor was displayed in yellow. Participants were asked to select the target using either a single channel input twice or a-coord input as quickly and accurately as possible. In the single channel condition, participants first selected the correct wedge using one channel (e.g. pressure or roll). Once the participants landed on the desired wedge, they could then move up to second dimension in the 2D menu by pressing the CTRL key with the non-dominant hand, and then applying the same channel again. In the a-coord input condition, participants selected the wedge using one channel (e.g. roll) and the target item using another channel (e.g. pressure). With a-coord input, simultaneous movement across both channels was possible. In both conditions, the final target selection was made by pressing a hardware button (CTRL key) using the non-dominant hand. To undo any action users could simply lift up the pen. Prior to the experiment, participants were shown the experimental setup, and were given several practice trials in each condition. For the a-coord input techniques, participants were shown how channels could be engaged simultaneously (e.g., applying pressure towards the target circle and rolling the pen towards the desired wedge, at the same time). However, participants were not required to engage in parallel action and could complete the task by allocating control to one channel and then the other. Breaks were enforced at the end of each block of trials. The entire experiment lasted approximately 30 minutes. Roll - For roll input, we defined the initial roll value of 0° as indicated in Figure 2. According to prior work, rolling under 10° was usually incidental and anything beyond ±90° is suboptimal [3]. Participants could roll the pen in either direction. Since our visual feedback consisted of a full circular layout, we employed a 1:2 mapping between the motor and visual space for roll. Tilt – For the tilt channel, we consider only tilt in the azimuth angles, where 0° was mapped to a tilt to the East as indicated in Figure 2. Combining these three channels, we get three different acoord techniques: Roll + Pressure (R+P) (Figure 3.a), Tilt + Pressure (T+P) (Figure 3.b) and, Tilt + Roll (T+R) (Figure 3.c), where the first channel moves along the first dimension (radially) and the second channel controls the cursor in the second dimension (linearly). We selected these visuomotor mappings based on prior work (as described in the section “Properties of Auxiliary Channels”).We included two baseline single-channel techniques: Pressure + Pressure (P+P) and Roll + Roll (R+R). Tilt + Tilt requires a different visual mapping, since tilt works best with radial feedback, which we did not want to restrict ourselves to. Therefore, we excluded Tilt + Tilt to avoid introducing potential confounds. Figure 3 – Three a-coord techniques we evaluated. Roll+Pressure (R+P); Tilt+Pressure (T+P); Tilt+Roll (T+R). Figure 2 – Visual feedback for 4 × 4 (left) and 8 × 8 (right) levels. The black arrowheads indicate the target wedge. Design As indicated above, to avoid a combinatorial explosion of different a-coord input styles, we restricted our study to We placed the target at 3 distances: 25%, 50%, and 75%, of the total input range for each channel, for both the first and second dimensions (Figure 2). Overall, the experiment employed a 5×2×3 within-subjects factorial design. The independent variables were Technique: P+P, R+R, R+P, T+P and T+R; Number of Levels per dimension (low, high, 4 and 8 levels in both dimensions respectively); and Target Distance (25%, 50%, and 75%). Technique was counterbalanced across participants using a Latin square, while the other factors were presented in random order. Each trial representing a Technique × Number of Level × Target Distance combination, was repeated 4 times by each participant. Given that our focus is on the feasibility of a-coord input as opposed to developing a novel 2D menu technique, we omit comparisons to techniques such as marking menus [9]. With respect to marking menus, we also wish to focus on contextual tasks, where pen movement is not involved. Results The data were analyzed using Repeated-Measures ANOVA and Bonferroni corrections for post-hoc comparisons. spite this, we find that a-coord is more efficient than using a single channel alone. As to be expected, there was a significant effect of Number of Levels on completion time (F1, 9 = 135.2, p < 0.001), with participants slower at 8 levels (4006 ms, s.e. 181) than at 4 levels (2661 ms, s.e. 104). This effect was generally consistent across techniques. There was no main effect of Target Distance on completion time (F2, 18 = 1.93, p = 0.17), however, the interaction effect between Technique and Target Distance was significant (F8, 72 = 6.15, p < 0.001). The nature of the interaction was difficult to interpret; however, it appears as though the poor performance of techniques involving pressure (P+P, R+P, and T+P) was mainly caused by the poor performance of those techniques when low pressure levels were required (targets at 25%). This is consistent with the findings from the prior work [8], showing that people have difficulty controlling pressure at its lower end. Task Completion Time Completion time measured the time from the target’s appearance to the time participants successfully selected it, including errors. The RM-ANOVA yielded a significant main effect of Technique (F4, 36 = 46.33, p < 0.001) on completion time. The means for each technique are displayed in Figure 4. Post-hoc comparisons showed the 3 dual-channel techniques (T+P: 2315 ms, s.e. 182l; T+R: 2830 ms, s.e. 185; R+P: 2841 ms, s.e. 179) were all significantly faster than the 2 single-channel techniques (R+R: 4338 ms, s.e. 127; P+P: 4341 ms, s.e. 231; p < 0.001). There was also a nonsignificant trend indicating that T+P was faster than R+P (p=0.065), but there was no difference between R+P and T+R (p = 1). The difference between the two singlechannel techniques was not significant (p = 1). For the single-channel techniques, completion time can be decomposed into two sequential target acquisition components: the time it takes to make a successful selection on the first level, and the time from the end of the first task to the end of the trial. For P+P, since pressure is unidirectional, there was an additional adjustment cost between the two task components, where participants had to release the pressure after the first task by lifting the pen tip, and to land down the pen again to start the second task (Figure 4.left). Figure 4 shows the task decomposition for each of the two single-channel combinations. We observe that participants require less time on the second invocation of the channel. This goes contrary to our expectations, in that the second invocation should take longer due to the mechanical readjustment of the finger after having invoked that channel once. We believe that this is still likely the case, but that users probably built muscle memory from the first phase, given that the targets were all laid out at the same distance in the second level. In retrospect, we created a condition that unintentionally favoured the single channel input. De- Figure 4 – Left: Task completion time shown by technique. Right: Error Rate shown by technique. (Error Bars represent ±1 s.e.) Number of Errors An error occurred if the participant selected the wrong target. For single channels, error was recorded only if the item on the second level was not selected properly. The trial did not stop until the proper target was selected. The RM-ANOVA yielded a significant main effect of Technique (F4, 36 = 4.47, p = 0.01) on error rate. Post-hoc analysis showed that T+R (5.4%, s.e. 0.9%) had significantly fewer errors than P+P (17.5%, s.e. 3%) (p=0.034). There were also non-significant trends indicating that T+R might be less error prone than R+R (11.2%, s.e. 1.9%, p=0.067) and T+P (20.6%, s.e. 4.6%, p=0.072). There was no significant difference between T+R and R+P (14.3%, s.e. 3.1%, p=0.220), nor were there significant differences between the remaining techniques (p=1). There were significant main effects of Numbers of Levels (F1, 9 = 35, p < 0.001) and Target Distance (F2, 18 = 1.93, p < 0.001) on error rate. Participants made twice as many errors with 8 levels (18.2% s.e. 1.8%) than they did with 4 levels (9.4% s.e. 1.8%). For target distances, there were significantly more errors with targets at 25% distance (23.1%, s.e. 3.3) than with targets at 50% distance (11.3%, s.e. 1.2%) and 75% (7%, s.e. 1.5%) (p < 0.05). Post-hoc analysis showed no significant difference between the 50% and 75% distances (p = 0.1). Finally, there was a significant Technique × Target Distance interaction effect (F8, 72 = 0.07, p < 0.05). Similar to our results for completion time, the interaction was at least partly due to the techniques involving pressure, where the error rate decreased rapidly as the target distance increased. Discussion A-Coord Input Performance Our results reveal several trends. For all a-coord input styles tested, users were faster than using an auxiliary channel twice. Based on our results across all our measures, Tilt+Roll afforded the best overall result, with completion times below those of the single channels, and error rates in an acceptable range. One primary reason is that Tilt does not require users to traverse a range of item before reaching the target (Table 1). Additionally, Roll can control a larger number of items than pressure, thus disadvantaging this latter channel (Table 1). While Tilt+Pressure showed a trend towards being the fastest technique, it also exhibited a high error rate, making it perhaps the least desirable technique of all three a-coord styles we tested. The fact that Tilt takes considerably less time to stabilize than either roll or pressure is to be expected due to the nonsequential nature of acquiring items through tilt-azimuth. Users take roughly 22% of the total task time to operate and stabilize tilt. This corresponds to a value between 700 and 850 msecs, which matches very closely to tilt performance when operated alone, as shown in earlier work [22]. Input with the second channel, i.e. Roll or Pressure with Tilt, takes approximately 75% of the total task time (i.e. users seem to take the remaining 25% of total task time to select the target with the button using the non-dominant hand). With Roll+Pressure, we see users on average operate Roll at 50%, and Pressure at 72% of total task time. These results indicate that users stabilized the first channel before proceeding to the final goal. They may also suggest that the channel with more controllable input range (Table 1), i.e. in this case Roll or Tilt, gets stabilized before the one with less control. Error Rates Error rates are similar to the ranges found in earlier studies on single channel input (see [3, 16, 22]). These range between 5% and 20%. Such errors can be minimized with better discretization functions [8] and by using fewer number of items [22]. Other improvements can be found when users are trained and improve with learning [8]. Extending the Number of Controllable Items Our results show that any A-coord technique with 4×4 items has a comparable performance with other single channel techniques. These results show that we can extend the range of discrete items that was previously possible with single auxiliary channels. We see that a-coord input facilitates a factor of 2 to 3 times the possible range with single channels. Even with a conservative extension, of up to 4×4 items, error rates across a-coord input are within the bounds of what was previously reported with single channels alone. Figure 6 – Average percentage of time consumed by each channel over the length of a trial. We further examine the performance of the non-leading channel (i.e. the channel which stabilized last) for the period in which both channels operate simultaneously. For example, during the period it takes Tilt to stabilize (22% of the overall task time in Tilt+Roll or roughly 700 msecs, represented by the red vertical bar in Figure 7) we observe several trends. With Tilt+Roll we find that while users are operating Tilt, the values of Roll also grow linearly and this continues even after Tilt gets stabilized. In the case of Tilt+Pressure and Roll+Pressure, the non-leading channel Pressure is controlled in a log manner. This suggests that during the period that both channels are operating, pressure quickly ramps up and then slows down after the leading channel’s becomes steady. Coordination We examine the amount of coordination facilitated by acoord input by breaking down the total completion time by the amount of control exhibited by each individual channel (Figure 6). We observe a few trends. First, we notice that while users still operate both channels in conjunction, they tend to stabilize one channel before completing the task with the other. This result goes contrary to our initial expectation that both channels would always be operated together, instead of one leading the other. Furthermore, stabilizing one channel before the other might explain the improved efficiency and error rates we observed with certain a-coord styles. For example, users stabilize Tilt very quickly, which may explain why combinations with this channel, such as Tilt+Roll, worked better than other techniques. Figure 7 – Degree of control with the non-leading channel until the leading channel stabilizes, i.e. stops changing. The red vertical bar represents the timestamp at which the leading channel stabilizes. Left: with Tilt+Roll, Roll is controlled in a linear fashion; Middle, Right: non-leading channel Pressure, is controlled in a log manner. All R2 are above 0.9. Overall, these observations on channel coordination suggest that users tend to operate both channels conjunctively, within the time frame used for operating the leading channel. The conjunctive operation of a-coord input has the potential to yield performance gains in tasks other than 2D discrete item selection. We demonstrate how to extend this conjunctive operation to a different task in our next study. COORDINATING CHANNELS FOR MULTI-PARAMETER SELECTION AND MANIPULATION Our first study revealed that users can conjunctively coordinate two auxiliary channels. In our next study we explored this a-coord input feature through multi-parameter selection and manipulation, a task that involves continuous manipulation and has a more inherent two-step structure:. The common task of multi-parameter selection and manipulation requires users to first select a desired parameter before they can actually change its value. We adapt a-coord input such that users concurrently choose a parameter and manipulate it. This form of interaction would be suitable for users who know a priori the value of the target they wish to set a parameter at. In these situations, a-coord input could be used to select and manipulate the value of a parameter through a single and continuous action. We note that the pen’s auxiliary channels were designed for continuous and fluid input, such as for drawing. We therefore harness this natural design feature but in a multi-step fashion. One challenge in adapting a-coord input for a multiparameter selection and manipulation task is to avoid inadvertently setting values for parameters that were not selected. Figure 8 (left) shows how to adjust the value of multiple parameters, e.g. brightness or contrast of an image, with P+R. A user can move between sliders using pressure. Only the active slider will get highlighted, for which its value can be altered by rolling the pen. Users can press a key to confirm the change. With a-coord input, rolling the pen while pressing will unintentionally change the value of all sliders, active or inactive. To address this issue, we introduce a ghost wiper on every slider. Ghost wipers are semitransparent and work the same way as real wipers but without changing the value of the parameters. They only show the potential change of the value. When users press the selection key, the change takes place on the active slide, while all other sliders remain unchanged (Figure 8 left). As an add-on benefit, a-coord interfaces allow designers to hide the inactive sliders to save expensive screen real estate. For example the P+R interface could consist of stacked sliders with only the active one being visible. Similarly, the T+R interface only shows the slider associated with the pen’s tilt angle (Figure 1). EXPERIMENT 2 – EXPLOTING A-COORD INPUT FOR A TWO-STEP CONTINUOUS MANIPULATION TASK This study measures user performance of a-coord input in a multi-parameter selection and manipulation task. Unlike 2D discrete item selection, the two sub-tasks in a multiparameter selection and manipulation task are asymmetric, i.e. each channel plays a different role – one for discrete item selection and the other for continuous variable manipulation. The two-step process requires users to hold the leading channel steady while manipulating the non-leading channel, thus testing the users' ability to exhibit this type of control with a-coord input. An additional distinction between this task and the 2D selection is that manipulating a continuous variable requires finer control. We use only Roll for manipulating the continuous variable as our pilot studies showed that Pressure did not afford sufficient bidirectional control for fine-grain input, and Tilt did not map naturally to such a task. We thus mapped parameter selection to Pressure and Tilt. Finally, we were also interested in knowing if a-coord input affords a comparable performance to an existing multi-parameter selection and manipulation technique. We included the FaST Slider [12] as a baseline technique in the study. Other techniques exist (as described earlier) but FaST sliders have shown to be easier to learn, than for example, FlowMenus [6]. Participants and Apparatus Twelve right-handed participants (2 females) between the ages of 20 and 35 were recruited for this study. Participants had little or no experience with pen-based interfaces. We used the same apparatus as in Experiment 1. Tasks and Procedure For the a-coord techniques, participants were asked to select a desired slider using Pressure or Tilt, and then use Roll to adjust the position of the wiper to a target value shown by a vertical bar (Figure 8 left). The wiper was initially placed in the middle of the slider of 360 pixels (50.4 mm) high. Rolling the pen 1° in the counter-clockwise moved the wiper up by 1 pixel, and vice versa, providing 360 discrete levels and ensuring sufficient smoothness and continuity. When the wiper reached the target distance, participants pressed the CTRL key using the non-dominant hand to confirm a selection. With the FaST Slider, participants first selected a desired slider using a marking menu [9]. The slider appeared at the position where the participants lifted the pen (Figure 8 right). They then used the pen tip to drag the wiper to the target value, pressing the CTRL button to confirm selection. The height of the entire slider widget remained the same for all techniques. Figure 8 – (a) Using pressure to select a desired slider, and using Roll to adjust the position of the wiper. (b) FaST Slider. A trial ended when participants successfully changed the desired parameter to the target value. Prior to the study, participants were given practice trials to familiarize themselves with all techniques. Similar to Experiment 1, they were shown how to engage in a-coord input in a coordinated manner, but this was not enforced in the study. Design The experiment employed a 3×2×2×3 within-subjects factorial design. The independent variables were Technique: P+R, T+R, and FaST Slider; Number of Parameters: Low (4) and High (6); Granularity: Coarse-grain, Fine-grain; and Target Distance: Near, Mid, and Far. T+R was always faster than P+R and FaST Sliders, but that differences between the latter two were more nuanced. In some conditions (e.g. coarse-grained and 4 parameters), P+R showed a comparable performance with FaST Slider. Number of Parameters – High was set to 6 items since results from the first study showed that pressure was hard to control with 8 levels. Granularity – we used wipers of 2 different sizes to adjust the level of granularity. For the fine-grain setting, we used a wiper of 15 × 30 pixels (4.5 × 8.4 mm), and for the coarse-grain setting, we used a wiper of 30 × 30 pixels (8.4 × 8.4 mm). Target Distance - we randomly placed the target within 3 intervals: Near (10%-30%), Mid (40%-60%), and Far (70%-90%), of the total input range. For rolling, the direction of roll was randomly chosen for each of the 3 target distances. In other words, distance Near could be randomly interpreted to be between ±(9°-27°). Technique - was counterbalanced across participants using a Latin square, while the other factors were presented in a random order. The study consisted of four blocks, each consisting of 2 trials. There were 3 Techniques × 2 Numbers of Parameters × 2 Granularities × 3 Target Distances × 4 Blocks × 2 Repetitions × 12 Participants = 3456 trials in total. Results The data were again analyzed using Repeated-Measures ANOVA and Bonferroni corrections for post-hoc pair-wise comparisons. For the sake of brevity, we concentrate our reporting on our primary factor of interest, Technique. Task Completion Time RM-ANOVA yielded a significant effect of Technique (F2, 22 = 23.86, p < 0.001) on task completion time. The means for each technique are displayed in Figure 9. Post-hoc comparisons showed that T+R (1703 ms, s.e. 91) was significantly faster than FaST Slider (2219 ms, s.e. 88) and P+R (2339 ms, s.e. 106) (p < 0.001). The difference between FaST Slider and P+R was not significant (p = 1). Figure 10 – The interaction effects for completion time. Number of Errors The RM-ANOVA yielded a significant main effect of Technique (F2, 22 = 12.48, p < 0.001) on the number of errors (Figure 9). Post-hoc analysis showed that P+R (12.1%, s.e. 1.7%) had significantly more errors than T+R (4.5%, s.e. 1%) and FaST Slider (6.3%, s.e. 1.5%) (p < 0.05). There was no significant difference on between T+R and FaST Sliders (p = 0.82). There was, however, a significant Technique × Target Distance interaction (F4,44 = 22.03, p < 0.001), indicating that the difference between P+R and the other two techniques occurred mainly at Low target distances, where P+R was more error prone. DISCUSSION Experiment 2 Results Results of experiment 2 show that a-coord input can be applied to a task involving continuous manipulation and a more distinct two-step process than the discrete item selection task studied in experiment 1. Of the techniques evaluated, combining Tilt+Roll led to the lowest completion times and was comparable to an existing technique, FaST Sliders, in terms of errors. We also note that holding a Tilt value while rolling was more controllable than holding a certain Pressure value. Although the results showed that combining Tilt and Roll was superior to combining Pressure and Roll for a task of this nature, the latter combination can still have a comparable performance with a careful design, e.g. few discrete items (i.e., number of parameters) for pressure and coarse-grained control for rolling. Utility of A-Coord Input We summarize the primary findings from our two studies as follows:  A-coord input provides a larger input range than what Figure 9 - Left: Task completion times. Right: Error rates. In addition to the above main effect, there were significant interactions between Technique × Number of Parameters (F2, 22 = 22.79, p < 0.001), Technique × Granularity (F2, 22 = 4.89, p = 0.01), and Technique × Target Distance (F4, 44 = 5.25, p = 0.001) (Figure 10). These effects demonstrate that is available with single channels (up to 64 or 8×8 items vs. 16 items, table 1);  Some a-coord styles are more efficient than combining single channels serially;  Of the combinations studied, Tilt+Roll affords the best a-coord control for discrete 2D target selection;  Tilt+Roll a-coord input is efficient for parameter selection and manipulation, and is a viable alternative or addition to existing techniques such as FaST sliders. Expanding on these primary findings, our experiments show that a-coord input is a viable input technique across two qualitatively different tasks. Experiment 1 showed that a-coord input can extend the bandwidth available through a single auxiliary channel as channels are operated in conjunction. On a conservative side the bandwidth with acoord input easily extends to 16 items (4×4), with much higher levels possible for techniques that don’t involve Pressure. Our results also indicate that when combining two channels, users have a tendency to stabilize one channel first - either Tilt, which is stabilized rapidly, or Roll. This tendency fits with other research on parallel control, which found that users will not always allocate control equally when operating a multiple degrees-of-freedom input device, particularly when equal allocation results in a biomechanically awkward motion [11]. However, we also observed that while both channels are in operation, users exhibit a high amount of parallelism. This latter result suggests that coordinating multiple auxiliary channels on the pen is a relatively natural motion. In experiment 2, we demonstrated that a-coord input is applicable to a more continuous task and one with a more distinct division of responsibilities for the channels. This latter task characteristic adds the challenge of having to hold the leading channel steady while operating the second channel. Our results indicated that Tilt and Roll was particularly effective for this multi-parameter selection and manipulation task, with performance either exceeding or comparable to an alternative technique (FaST Sliders). A loose correlation of our results to the comparative analysis of the techniques summarized in Table 1, may suggest the following. When an auxiliary channel has a rapid access method to discrete items, such as with Tilt, this channel stabilizes quicker in a-coord style interaction. However, in the process of stabilization, the channels involved are still being coordinated conjunctively. A second observation is in the amount of control possible and mapping of a task to a channel. For example, Roll was relegated to continuous parameter manipulation. This seems intuitive but also worked well since rolling affords a high degree of control. Application Scenarios Single channel input on the pen has been primarily proposed for in-context interactions [22], where the user does not need to move the pen for interaction. We feel that the acoord input enhances interactions in these types of scenarios but further facilitates input with the pen in ways that were not easily captured with single channels alone. We present three application scenarios: 2D menus, improved stimulus-response compatibility, and integral actions. 2D Menus - With a-coord input, contextual menus take on a new dimension. The technique can, increase the range and number of items selectable and allow for 2D organization of menu items. Tilt menus have been shown to work well for menu selection [22], but are limited in range. In our case, using even a conservative technique, (pressure and tilt) we can immediately get 16 items for selection (Figure 1.a), with much larger item sets possible, such as for the color palette (shown in Figure 11.a), with Tilt and Roll. Beyond simply extending the number of items, a-coord input also permits 2D menu organization, with the structure (e.g., number of items per level) tailored to the strengths of the specific dual-channel combination. We don’t necessarily need to constrain menus to one level before showing the next - we can show all menu items and then allow the user to simply move to their target in a more parallel manner. Figure 11 - Sample applications that can benefit from a-coord input, as per our results. (a) Tilt-Roll 2D color palette; (b) Tilt-Roll 3D-manipulation; (c) Tilt-Roll Volumetric data navigation ; (d) Pressure-Roll dynamic CD ratio adjustment. Extended Stimulus-Response Compatibility – A-coord input can also enhance stimulus-response compatibility. For example, Tilt can easily be used to select a mode (i.e. 2-6 modes) before engaging in a rolling action. Examples of combining channels in this manner include common transformations in 3D applications: we can map Tilt to axis selection, and Roll to manipulation; the latter could rotate or scale an object along the axis selected by Tilt (Figure 11.b). This can replace the use of small handles commonly used for such tasks in graphics applications, which are prone to parallax issues with the pen. Volumetric data navigation often requires users to change the viewing angle of a virtual camera while manipulating the camera’s depth. Tilt and Roll can provide a smoother control, for changing the orientation of the clipping plane (Tilt) and simultaneously the depth of the plane (Roll) (Figure 11.c). Integral Actions - A-coord input can also support integral actions, ones that map well to the perceptual structure of the task [7]. For example to fine-tune rotation of an object, Pressure can be mapped to controlling the CD ratio while Roll rotates the object (Figure 11.d). These sub-actions could be easily carried out simultaneously to enhance fluidity with contextual actions. CONCLUSION AND FUTURE WORK In this paper, we investigate the benefits of the design space of a-coord input through two experiments, which systematically studied several fundamental questions of such an input style. The results of the first study show that a-coord input can effectively improve the bandwidth of the pen’s auxiliary channels with high efficiency and accuracy and by operating the channels in parallel for at least some duration of the task. These findings can be applied to a continuous two-step task. We caution, however, that our understanding of a-coord input is still in its infancy. Additional empirical work is required to: (i) identify more precise usable ranges for each channel combination; (ii) determine the generalizability of our results across all other channel combinations; (iii) empirically verify the value of a-coord input in some of the application scenarios we have proposed; (iv) explore acoord input for direct settings; (v) investigate the effect of a-coord input for different visual mappings; and (vi) determine users’ qualitative responses to a-coord input in comparison to other alternatives. Answers to some of these pertinent questions along with our findings can make acoord input a reliable, effective and common interaction method for pen-based interfaces. ACKNOWLEDGEMENTS We acknowledge NSERC for providing the funds, our study participants and our lab mates, David McCallum and Cary Williams for assisting with the video. REFERENCES 1. Balakrishnan, R. and Hinckley, K. (2000). Symmetric bimanual interaction. In CHI’00. 33-40. 2. Buxton, B., Wired Magazine, 7/26 2011, http://www.wired.com/magazine/2011/07/st_thompson_ breakthrough/, accessed January 2012. 3. Bi, X., Moscovich, T., Ramos, G., Balakrishnan, R. and Hinckley, K. (2008). An exploration of pen rolling for pen-based interaction. In UIST’08. 191-200. 4. Cechanowicz, J., Irani, P. and Subramanian, S. (2007). Augmenting the mouse with pressure sensitive input. In CHI’07. 1385-1394. 5. Grossman, T., Hinckley, K., Baudisch, P., Agrawala, M. and Balakrishnan, R. (2006). Hover widgets: using the tracking state to extend the capabilities of pen-operated devices. In CHI’06. 861-870. 6. Guimbretiére, F. and Winograd, T. (2000). FlowMenu: combining command, text, and data entry. In UIST’00. 213-216. 7. Jacob, R. J. K., Sibert, L. E., McFarlane, D. C. and Jr, M. P. M. (1994). Integrality and Separability of Input Devices. TOCHI, 1 (1), 3-26, 1994. 8. Kang, S., Irani, P., Gustafson, S. and Subramanian, S. (2008). PressureFish: a method to improve control of discrete pressure-based input. In CHI’08. 1295-1298. 9. Kurtenbach, G. P., Sellen, A. J. and Buxton, W. A. S. An empirical evaluation of some articulatory and cognitive aspects of "marking menus”, HCI, 8(1), 1-23 1993. 10. Li, Y., Hinckley, K., Guan, Z. and Landay, J. A. (2005). Experimental analysis of mode switching techniques in pen-based user interfaces. In CHI’05. 461-470. 11. Masliah, M. R. and Milgram, P. (2000). Measuring the allocation of control in a 6 degree-of-freedom docking experiment. In CHI’00. 25-32. 12. McGuffin, M., Burtnyk, N. and Kurtenbach, G. (2002). FaST Sliders: Integrating Marking Menus and the Adjustment of Continuous Values. In GI’02, 35-42. 13. Mizobuchi, S., Terasaki, S., Keski-Jaskari, T., Nousiainen, J., Ryynanen, M. and Silfverberg, M. (2005). Making an impression: force-controlled pen input for handheld devices. In CHI’05. 1661-1664. 14. Ramos, G. and Balakrishnan, R. (2003). Fluid interaction techniques for the control and annotation of digital video. In UIST’03. 105-114. 15. Ramos, G. and Balakrishnan, R. (2005). Zliding: fluid zooming and sliding for high precision parameter manipulation. In UIST’05. 143-152. 16. Ramos, G., Boulos, M. and Balakrishnan, R. (2004). Pressure widgets. In CHI’04. 487-494. 17. Ramos, G. A. and Balakrishnan, R. (2007). Pressure marks. In CHI’07. 1375-1384. 18. Ren, X., Yin, J., Zhao, S. and Li, Y. (2007). The Adaptive Hybrid Cursor: A Pressure-Based Target Selection Technique for Pen-Based User Interfaces. In INTERACT’07. 310-323. 19. Song, H., Benko, H., Guimbretière, F., Izadi, S., Cao, X. and Hinckley, K. (2011). Grips and gestures on a multi-touch pen. In CHI’11. 1323-1332. 20. Suzuki, Y., Misue, K. and Tanaka, J. (2007). Stylus Enhancement to Enrich Interaction with Computers. In HCII’07. 133-142. 21. Tian, F., Ao, X., Wang, H., Setlur, V. and Dai, G. (2007). The tilt cursor: enhancing stimulus-response compatibility by providing 3d orientation cue of pen. In CHI’07. 303-306. 22. Tian, F., Xu, L., Wang, H., Zhang, X., Liu, Y., Setlur, V. and Dai, G. (2008). Tilt menu: using the 3D orientation information of pen devices to extend the selection capability of pen-based user interfaces. In CHI’08. 1371-1380. 23. Xin, Y., Bi, X. and Ren, X. (2011). Acquiring and pointing: an empirical study of pen-tilt-based interaction. In CHI’11. 849-858.