High Latency Unmanned Ground Vehicle Teleoperation Enhancement by Presentation of Estimated Future through Video Transformation

Abstract

Long-distance, high latency teleoperation tasks are difficult, highly stressful for teleoperators, and prone to over-corrections, which can lead to loss of control. At higher latencies, or when teleoperating at higher vehicle speed, the situation becomes progressively worse. To explore potential solutions, this research work investigates two 2D visual feedback-based assistive interfaces (sliding-only and sliding-and-zooming windows) that apply simple but effective video transformations to enhance teleoperation. A teleoperation simulator that can replicate teleoperation scenarios affected by high and adjustable latency has been developed to explore the effectiveness of the proposed assistive interfaces. Three image comparison metrics have been used to fine-tune and optimise the proposed interfaces. An operator survey was conducted to evaluate and compare performance with and without the assistance. The survey has shown that a 900ms latency increases task completion time by up to 205% for an on-road and 147% for an off-road driving track. Further, the overcorrection-induced oscillations increase by up to 718% with this level of latency. The survey has shown the sliding-only video transformation reduces the task completion time by up to 25.53%, and the sliding-and-zooming transformation reduces the task completion time by up to 21.82%. The sliding-only interface reduces the oscillation count by up to 66.28%, and the sliding-and-zooming interface reduces it by up to 75.58%. The qualitative feedback from the participants also shows that both types of assistive interfaces offer better visual situational awareness, comfort, and controllability, and significantly reduce the impact of latency and intermittency on the teleoperation task.

Appendix: Incorporating Control Input Delay with Visual Latency

Latency into the simulation platform can be achieved either by adding delay to the visual feed or by delaying the control input to the simulation platform. The authors of the paper are confident that the impacts of either of these delay factors, or a combination of them, are perceived as the same from the perspective of a teleoperator. This appendix discusses the methods through which the control input delay can be achieved and the impacts on a teleoperator.

In the main sections of the paper, latency was achieved purely by delaying the visual feed. To achieve the visual delay, a single computer unit was used to connect the simulating platform with controller devices, visual feed receiver camera, and the latency control and teleoperation enhancement Simulink® model. However, to incorporate control input delay along with the visual delay, we have used another personal computer (PC) unit, referred to as PC-2 in the rest of this section. The system diagram of the modified teleoperation simulation system has been provided in Fig. 13. In this modified system, the controller devices (steering wheel, brake and acceleration pedals) are connected to the PC-2. The vehicle simulation platform (game engine), the visual feed receiver, and teleoperation enhancement model is hosted by the previously used PC, named PC-1 hereafter. The two computers can be connected either by local area network (LAN), wide area network (WAN), or Internet connection. For our case, the computers were connected to the university LAN.

Aside from these new amendments, the rest of the system is as described in Section 3.

Fig. 13

System diagram to incorporate both control and visual latency to the system. Here PC-1 hosts the simulation platform and the enhancement model, and PC-2 connects to the controller devices

To add latency between the two computers we have used a third party software called Clumsy. Clumsy uses the Windows Packet Divert library to stop, capture, lag, drop, or tamper with packets on a living network. Any amount of system to system latency is achievable using this tool. An example of varied latency between two computers using Clumsy is shown in Fig. 14. To connect and receive control input from PC-2 to PC-1 we have used another third party software called VirtualHere. Although USB devices usually need to be directly connected to a computer to be used, VirtualHere facilitates the transmission of USB signals over a LAN, WAN or Internet connection to a remote machine, allowing for virtual connection of USB devices over a network.

Fig. 14

PC-to-PC packet delay (any amount) facilitated by the WinDivert based Clumsy tool

To investigate the impact of control latency we have run teleoperation sessions on both on-road and off-road tracks. These are the same tracks used by the survey participants. We have compared the outcomes with the non-assisted delayed visual feed outcomes by the participants (From Table 3). The non-assisted delayed feed is affected by the visual delay only. The comparison is in Table 5 below. For both the on-road and off-road tracks, the average speed and time taken to complete one lap is very similar. A small amount of difference exists as the control only delay is a little higher than that of the visual delay.

Table 5 Comparison of Control delay only and visual feed delay only teleoperation

Table 5 demonstrates that a certain amount of latency or delay in the teleoperation control loop, regardless of its source, will have the same impact on the teleoperator. To reinforce this statement and to prove the robustness of our teleoperation enhancement technique, we have conducted teleoperation runs on on-road and off-road tracks where the total delay in the loop consists of both control input delay and visual output delay. The outcome of the teleoperation sessions is provided in Table 6. For the on-road teleoperation session, the cumulative latency was 1200 ms where the control input latency was 560 ms and the visual feed latency was 640 ms. Without any video transformation based assistance it took 499 s to complete the 2 km track with an average speed of 14.42 km/h. Using our sliding window enhancement, the completion time reduced to 428 s with an average speed of 16.8 km/h. Using the sliding and zooming window both the time and speed improved further. Incidents of oscillation and out of track events also improved substantially using our assisted windows.

Table 6 Comparison of teleoperation performance affected by combined latency

For the off-road track, the figures correspond with the previous numbers. In this case, the total latency is 1076 ms where 560 ms latency is induced from the controller to PC-2 and 516 ms latency is induced by our Simulink® visual transformation model. Using the sliding window the completion time reduced from 385 s to 282 s. For sliding and zooming the time is further reduced. The oscillation and out of track events reduced to zero for both the sliding only and sliding with zooming windows. Table 6 reflects the same outcome to that obtained via the participants’ survey. It reconfirms the findings of our research that our video transformation-based assisted windows enhance an operator’s performance for high latency ground-vehicle teleoperation. Based on these results, it can confidently be stated that whether the delay is present only in the visual feed, only in the control input, or a combination of the two (as would be the case in a real-world scenario) the impact is effectively the same from the operator perspective. This validates the platform that has been setup to evaluate the teleoperation enhancement techniques.