-
Background
This paper investigates autonomous robotic experience data collection and application across large-scale and open environments. Current robotic learning methods provide solutions for acquiring individual skills, but face significant challenges when it comes to generalizing over an infinite number of tasks and diverse settings. Current robotic datasets have been collected in controlled lab settings and the amount of data needed in the real-world far exceeds what has been collected. -
Existing Work Existing works are mostly conducted in limited lab environments, depending on a large amount of human-assisted data collection, which greatly restricts the diversity and scale of data. Automated data collection must be able to cover different environments and operate with limited human supervision.
- Introduced a system called AutoRT
-
Challenge 1: How to scale up the deployment of operational robots to collect data in completely unseen scenarios? The AutoRT system uses existing foundation models to operate robots in nearly unsupervised conditions for completely unknown scenarios. By integrating Vision-Language Models (VLMs) for scene understanding and generating diverse and innovative instructions from Large Language Models (LLMs), AutoRT significantly scales up data collection for robot learning, while reasoning about autonomy trade-offs and safety constraints.
-
Challenge 2: How to ensure the data collected is reliable and can enhance robot generalization capabilities? AutoRT implements this by suggesting tasks autonomously and collecting over 77k robot operation examples in real-world buildings. The data collected with AutoRT is not only highly diverse but can align robot behavior with human preferences through prompting and critiquing, as well as the use of a "robot constitution".
-
After an extensive real-world evaluation over 7 months and across 4 different office buildings with a fleet of over 20 robots, the AutoRT system has gathered the results of 77,000 real-world robot trials, including both teleoperation and autonomous execution. AutoRT scales robot deployment allowing one person to supervise 3-5 mobile manipulators. Evaluations studied how AutoRT can collect highly diverse data, be instructed to collect task-appropriate data, and demonstrated that such data can be used to improve state-of-the-art robot learning models. AutoRT also introduced methods to align robot behavior with human preferences using prompting and specific critiques.
The paper describes a system named AutoRT that uses large foundation models to control real-world robots to autonomously navigate and perform tasks. It marks the first instance of LLM-controlled robots operating autonomously in real-world settings, proposing their own goals, and taking actions toward those goals. The data collected by AutoRT is not only diverse but can improve the performance of robot learning models and be aligned with human preferences.