Explore or exploit: How our brains make choices

UNM research reveals the complex communications between neurons in primate brains

By Natalie Rogers

We make countless choices every day. It may be a difficult, complex choice—whether to take the job in a new city or stay in a current position—or be as simple as choosing between visiting a new restaurant or going to an old favorite. Decision-making shapes our knowledge and our perceptions of the world around us.

These simple and complex choices are generally between something familiar—where we mostly know what to expect—and trying something new, where the outcome might be unknown to us. But what happens inside our brains when we make one choice or the other?

Dr. Jeremy Hogeveen, Assistant Professor in the Department of Psychology, details his search for answers in a new article published in the prestigious neuroscience journal Neuron. The research was a collaboration between The University of New Mexico and Oregon Health & Science University (OHSU), and involved looking at modeling data of rhesus macaques, and observational data of humans using functional Magnetic Resonance Imaging (fMRI) technology.

The article, “The neurocomputational bases of explore-exploit decision-making,” looks at activity in the brain when primates and humans are presented with a choice between one option that maximizes information (explore) and one that prioritizes a known reward (exploit).

“This is the ‘explore-exploit tradeoff,’ and despite exploration being essential for healthy decision-making—a long-standing mystery in the field is why an animal might choose to explore something new and forego a familiar reward,” explains Hogeveen. “We were hoping to determine the brain computations that help motivate explore-exploit decisions.”

By “brain computations,” Hogeveen is referring to the calculations our brains make through a complex system of neurons, which communicate like circuits on a computer’s motherboard. The team is asking—what is the level of tolerance for uncertainty? Do these brain circuits value an immediate and known reward, or do they prefer to activate motivation to investigate something new?

Using computer modeling of explore-exploit decision-making processes through simulated tests, the team established that for both human adults and rhesus macaques, the estimation of future benefits motivates the choice to try the new options. The multi-disciplinary model, called a POMDP (partially observable Markov decision process) model, can assess numerous real-world situational decision-making processes. 

The next step was to collect fMRI data from macaques at OHSU, and humans at UNM and the Mind Research Network as they performed decision-making tasks. The subjects were presented with three choices, with limited opportunities to learn the value of each choice, and a random replacement of one of the choices over time. Uncertainty always accompanied the new choice because its reward value could not be predicted, and the only way to learn about the choice was to explore it.

Using these methods, Hogeveen and his team were able to merge the computational model with the fMRI data to visualize a partially overlapping, partially distinct network that communicated and calculated the value of exploring versus exploiting, effectively mapping out the decision-making processes of the entire brain.

“Interestingly, this involved distinct encoding of different value signals in brain regions that are highly evolved in primates like the lateral prefrontal cortex, which were more activated during exploration than exploitation,” Hogeveen says.

Basically, newer areas of the brain (evolutionarily speaking) such as the lateral prefrontal cortex, activated more when an unfamiliar choice was made while older brain systems including the amygdala and the obiotofrontal cortex showed increased activation while weighing the value of explorations vs. exploitation. This suggests that both the newer and older circuits in the brain worked together during the decision-making, process rather than against each other as was previously thought.

Hogeveen and his team took advantage of the research and data analysis resources at the UNM Center for Advanced Research Computing (CARC) for this project. CARC was instrumental in providing time-saving assistance, including the ability to run multiple datasets at once rather than one at a time. By taking advantage of CARC's parallel computing capabilities (available to all UNM researchers), Dr. Hogeveen was able to obtain his results in just over a day instead of the 10 - 25 days it would have taken if the data had been run on a single, stand-alone computer.

This project has promising future paths in cross-species primate research as well as bench-to-bedside interventions in clinical settings. In addition, according to Hogeveen this kind of explore-exploit decision-making research is crucial for understanding and treating brain disorders with circuit-specific pharmacueticals or transcranial neural stimulation when decision-making process may be compromised—in people with traumatic brain injuries, for example. But to understand how impaired brains work, scientists first must understand how typical brains function.

“Explore-exploit decision-making is impaired in brain disorders that have a significant public health impact,” says Hogeveen. “In my view and based on the thinking of lots of others in the field, overly-exploratory and overly-exploitative decision-making traits are likely to play a role in the challenges faced by many patients with psychiatric, neurological, or neurodevelopmental disorders.”

The team included Teagan Mullins, John Romero, and Elizabeth Eversole from UNM; Dr. Andrew Mayer from the Mind Research Network; Kimberly Rogge-Obando of Vanderbilt University (and former UNM undergraduate); and Dr. Vincent Costa from OHSU. This research continues to expand, and Hogeveen is hopeful for more knowledge from future work that will help assess and treat mental health challenges.