CNS*2020 Online has ended
Welcome to the Sched instance for CNS*2020 Online! Please read the instruction document on detailed information on CNS*2020.
Back To Schedule
Sunday, July 19 • 8:00pm - 9:00pm
P142: Using reinforcement learning to train biophysically detailed models of visual-motor cortex to play Atari games

Log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Haroon Anwar is inviting you to a scheduled Zoom meeting.

Topic: Haroon Anwar's Personal Meeting Room

Join Zoom Meeting

Meeting ID: 266 485 1890
Passcode: 6H7hTB

  Haroon Anwar, Salvador Dura-Bernal, Cliff C. Kerr, George L. Chadderdon, William W Lytton, Peter Lakatos, Samuel A. Neymotin
Computational neuroscientists build biophysically detailed models of neurons and neural circuits primarily to understand the origin of dynamics observed in experimental data. Much of these efforts are dedicated to match ensemble activity of the neurons in the modeled brain region while often ignoring multimodal information flow across brain regions and associated behaviors. Although these efforts have led us to improved mechanistic understanding of electrophysiological behavior of diverse types of neurons and neural networks, these approaches fall short of linking detailed models with associated behaviors in a closed-loop setting. In this study, we bridged that gap by developing biophysically detailed multimodal models of brain regions involved in processing visual information, generating motor behaviors and making associations between visual and motor neural representations by deploying reward-based learning mechanisms. We build a simple model of visual cortex receiving topological inputs from the interfaced Atari-game 'pong' environment (provided by the OpenAI’s Gym). This modeled region processed, integrated and relayed visual information about the game environment across the hierarchy of higher order visual areas (V1/V2->V4->IT). As we moved from V1 to IT, the number of neurons in each area decreased whereas the synaptic connections increased. This feature was included in the model to reflect the anatomical convergence suggested in the literature and to have a broader tuning for input features in progression up the visual cortical hierarchy. We used compartmental models of both excitatory and inhibitory neurons interconnected via AMPA (for excitation) or GABA (for inhibition) synapses. The strengths of synaptic connections were adjusted so that the information was reliably transmitted across visual areas. In our motor cortex model, neurons associated with a particular motor action were grouped together and received inputs from all visual areas. For the game Pong, we used two populations of motor neurons, for generating “up” and “down” move commands. All the synapses between visual and motor cortex were plastic, so that the connection strengths could be increased or decreased via reinforcement learning. When an action was generated in the model of motor cortex driven by visual representation of the environment in the model of visual cortex, that action generated a move in the game, which in turn updated the environment and triggered a response to the action: reward (+1), punishment (-1) or no-response (0). These signals drove the reinforcement learning at the synapses between visual cortex and motor cortex by strengthening or weakening them so that the model could learn which actions were rewarding in a given environment. Here we present an exploratory analysis as a proof-of-concept for using biophysically detailed modeling of neural circuits to solve problems that have so far only been tackled using artificial neural networks. We aim to use this framework to further simplify to make it more deep-learning-like and and also to extend the architecture to make it biologically realistic. Comparing the performance of trained models using different architectures will allow us to dissect the mechanisms underlying production of behavior and will bridge the gap between the electrophysiological dynamics of neural circuits and associated behaviors.


Haroon Anwar

Nathan Kline Institute for Psychiatric Research

CNS202 pdf

Sunday July 19, 2020 8:00pm - 9:00pm CEST
Slot 19