Computer Science > Machine Learning
[Submitted on 20 Jan 2022 (v1), last revised 21 Jan 2022 (this version, v2)]
Title:Safe Deep RL in 3D Environments using Human Feedback
View PDFAbstract:Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans - whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.
Submission history
From: Matthew Rahtz [view email][v1] Thu, 20 Jan 2022 10:26:34 UTC (31,623 KB)
[v2] Fri, 21 Jan 2022 16:10:14 UTC (39,643 KB)
Ancillary-file links:
Ancillary files (details):
- Cliff edge environment/images/image1.mp4
- Cliff edge environment/images/image10.mp4
- Cliff edge environment/images/image11.mp4
- Cliff edge environment/images/image12.mp4
- Cliff edge environment/images/image2.mp4
- Cliff edge environment/images/image3.mp4
- Cliff edge environment/images/image4.png
- Cliff edge environment/images/image5.mp4
- Cliff edge environment/images/image6.mp4
- Cliff edge environment/images/image7.png
- Cliff edge environment/images/image8.png
- Cliff edge environment/images/image9.mp4
- Cliff edge environment/instructions.html
- Dangerous blocks environment/images/image1.mp4
- Dangerous blocks environment/images/image10.mp4
- Dangerous blocks environment/images/image11.png
- Dangerous blocks environment/images/image12.mp4
- Dangerous blocks environment/images/image2.png
- Dangerous blocks environment/images/image3.mp4
- Dangerous blocks environment/images/image4.png
- Dangerous blocks environment/images/image5.mp4
- Dangerous blocks environment/images/image6.mp4
- Dangerous blocks environment/images/image7.mp4
- Dangerous blocks environment/images/image8.mp4
- Dangerous blocks environment/images/image9.mp4
- Dangerous blocks environment/instructions.html
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.