Seamless communication between humans and robots is a critical condition for successful human-robot interaction; not only for safety purposes but to properly convey information, coordinate activities, and minimize the cost of collaborative actions. In these human-robot scenarios, humans are required to act as critics and communicate the decisions via an input device (i.e. keyboard, mouse), which alienates them (albeit briefly) from the real goal of the task. Our research focuses on using specific brain signals that respond to success or failure events in order to inform the other agents of such outcomes so that these can adapt their strategies and, in the long term, learn the optimal behavior.
Note: see Supervising Robots with Muscle and Brain Signals for the integration of this project with gesture-based control via muscle signals.
Publication in ICRA 2017 (EEG only)
Authors: Andres F. Salazar-Gomez, Joseph DelPreto, Stephanie Gil, Frank H. Guenther, and Daniela Rus
Abstract: Communication with a robot using brain activity from a human collaborator could provide a direct and fast feedback loop that is easy and natural for the human, thereby enabling a wide variety of intuitive interaction tasks. This paper explores the application of EEG-measured error-related potentials (ErrPs) to closed-loop robotic control. ErrP signals are particularly useful for robotics tasks because they are naturally occurring within the brain in response to an unexpected error. We decode ErrP signals from a human operator in real time to control a Rethink Robotics Baxter robot during a binary object selection task. We also show that utilizing a secondary interactive error-related potential signal generated during this closed-loop robot task can greatly improve classification performance, suggesting new ways in which robots can acquire human feedback. The design and implementation of the complete system is described, and results are presented for real-time closed-loop and open-loop experiments as well as offline analysis of both primary and secondary ErrP signals. These experiments are performed using general population subjects that have not been trained or screened. This work thereby demonstrates the potential for EEG-based feedback methods to facilitate seamless robotic control, and moves closer towards the goal of real-time intuitive interaction.
PDF: available here
Publication in AURO 2020 (EEG+EMG) - see the project page here for more information
Authors: Joseph DelPreto, Andres F. Salazar-Gomez, Stephanie Gil, Ramin Hasani, Frank H. Guenther, and Daniela Rus
Abstract: Effective human supervision of robots can be key for ensuring correct robot operation in a variety of potentially safety-critical scenarios. This paper takes a step towards fast and reliable human intervention in supervisory control tasks by combining two streams of human biosignals: muscle and brain activity acquired via EMG and EEG, respectively. It presents continuous classification of left and right hand-gestures using muscle signals, time-locked classification of error-related potentials using brain signals (unconsciously produced when observing an error), and a framework that combines these pipelines to detect and correct robot mistakes during multiple-choice tasks. The resulting hybrid system is evaluated in a "plug-and-play" fashion with 7 untrained subjects supervising an autonomous robot performing a target selection task. Offline analysis further explores the EMG classification performance, and investigates methods to select subsets of training data that may facilitate generalizable plug-and-play classifiers.
PDF: available here
