TASC: Task-Aware Shared Control for Teleoperated Manipulation

We present TASC, a Task-Aware Shared Control framework for teleoperated manipulation that infers task-level user intent and provides assistance throughout the task. To support everyday tasks without predefined knowledge, TASC constructs an open-vocabulary interaction graph from visual input to represent functional object relationships, and infers user intent accordingly. A shared control policy then provides rotation assistance during both grasping and object interaction, guided by spatial constraints predicted by a vision-language model.

Our method addresses two key challenges in general-purpose and long-horizon shared control: (1) understanding and inferring task-level user intent, and (2) generalizing assistance across diverse objects and tasks. Experiments in both simulation and the real world demonstrate that TASC improves task efficiency and reduces user input effort compared to prior methods.

To the best of our knowledge, this is the first shared-control framework that supports everyday manipulation tasks with zero-shot generalization.

Overview of our TASC framework. The teleoperation loop begins with RGB-D observations of the scene, which are processed jointly with a VLM to construct an interaction graph encoding object centers, main axes, and functional relationships (blue: task-relevant, red: predicted target, grey: inactive). The inference module fuses this graph with motion cues to estimate a goal distribution, which guides the shared controller in blending user's translational input (Pos) with assistive rotations (Rot) for grasping and subsequent object interaction.

Pure Teleoperation v.s. TASC. Top row: Pure teleoperation without assistance | Bottom row: TASC-assisted teleoperation

TASC demonstrates zero-shot generalization across diverse everyday manipulation tasks

TASC handles complex scenarios involving multiple objects and sequential interactions

TASC adapts to various object configurations and spatial arrangements

BibTeX

@misc{fu2025tasc,
      title={TASC: Task-Aware Shared Control for Teleoperated Manipulation}, 
      author={Ze Fu and Pinhao Song and Yutong Hu and Renaud Detry},
      year={2025},
      eprint={2509.10416},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2509.10416}, 
}

TASC: Task-Aware Shared Control for Teleoperated Manipulation

Abstract

Highlights

Framework

Simulation on Robosuite

TASC in Real-World

Task Generalization

Multi-object Interaction

Different Objects Configuration

BibTeX