Project lead: Nancy Wang, Computer Science and Engineering
Advisors: Bing Brunton, Biology; Rajesh Rao, Computer Science and Engineering; Ali Farhadi, Computer Science and Engineering; Jeff Ojemann; Neurosurgery


There have been many successes in using experimental ECoG (Electrocorticography) and EEG (Electroencephalography) studies to decode brain activity and use them for brain controlled devices. These studies are particularly important for people who are motor challenged, for instance in cases of paralysis or amputation.

Images of a patient or patients in bed, superimposed over a graphHowever, the current state of research is very time consuming, tedious, and carefully, manually executed. In the future, to use brain interfacing technology in real life, we can’t rely on an expert to manually calibrate for each and every user. As well, each user will likely be doing many tasks outside the scope of the current one-dimensional experiments.

In our approach, we are collecting thousands of hours of simultaneous video (including a depth channel), audio and neurophysiological data. These are from patients undergoing clinical long term monitoring for epilepsy and they typically stay for about 1 week in the hospital room continuously with surgically implanted grids that record their neural activity in an intracranial fashion.

Because the patients are in the hospital continuously, they engage in the same kind of activities as at home, such as eating food, chatting with friends and family, watching TV and so on. Natural data like this has been rarely analyzed, particularly from human subjects.

Signals collected from humans “in the wild” are affected by a noisy environment and modulated by a complex range of behaviors. In our methods, we integrate these multiple streams of information with automated computer vision and speech recognition techniques to help tackle the noise and discover new brain states.

The overall goal of our research is to develop techniques for analyzing natural and long-term data, to discover new brain states and information about the brain that is normally missed in short task-based experiments and to develop methods to prepare for future long-term brain computer interfaces that can be used autonomously.