A review on exploration–exploitation trade-off in psychiatric disorders

Abstract

Balancing exploration and exploitation is a crucial aspect of adaptive decision-making, but psychiatric disorders can disrupt this balance in various ways, shedding light on their neurocognitive roots and guiding targeted interventions. In this systematic review, we aimed to delineate potential exploration–exploitation impairments across psychiatric disorders. Through a thorough search on PubMed, we identified forty-six relevant studies employing tasks probing exploration–exploitation balances, which we synthesized to reveal distinct patterns. These disorders are clustered into three categories: addictive patterns, emotional/cognitive disturbances, and neurological (neurodevelopmental and neurodegenerative) disorders. Our findings show that anxiety and mood disorders often enhance exploratory behaviors, while depression impact decision stability and reward sensitivity. In contrast, schizophrenia, OCD (Obsessive–Compulsive Disorder), and ADHD (Attention-Deficit/Hyperactivity Disorder) are characterized by excessive switching and difficulties in balancing exploration and exploitation, leading to impaired learning and adaptability. Additionally, disorders with addictive-like features disrupt optimal decision-making strategies by either heightening exploration or causing maladaptive persistence, thus skewing the balance away from effective decision-making. Individuals exhibiting addiction-like or compulsive behaviors often demonstrate imbalances in the explore-exploit trade-off, resulting in suboptimal decision-making characterized by reduced exploration, flawed foraging strategies, and impulsive or perseverative choices despite adverse outcomes. This suggests that such disorders may originate from dysfunctional foraging processes applied to decision-making. In sum, different patterns of exploration–exploitation balance in different disorders are crucial in understanding the difficulties in learning and decision making of neuropsychiatric disorders. This suggests that such disorders may stem from dysregulated decision-making processes, where uncertainty plays a central role. Dysfunctions in dopaminergic and noradrenergic pathways appear to disrupt the brain's representation of uncertainty, thereby altering exploratory behavior. In sum, the varying patterns of exploration–exploitation balance across different disorders are critical for understanding the challenges in learning and decision-making associated with neuropsychiatric conditions.

Introduction

Throughout life, humans are faced with countless decisions, ranging from simple choices about mundane matters to more complex dilemmas with serious implications. To thrive in dynamic environments with fluctuating rewards, individuals must skillfully balance exploiting known options with exploring uncertain alternatives. This delicate trade-off between exploitation and exploration represents a pivotal facet of mature decision-making [72]. In naturalistic foraging scenarios, rewards diminish as resources are consumed, necessitating decisions about when to leave depleted patches in search of new bounty. The omnipresent question is when to risk switching strategies to maximize one's rewards [23, 62]. Thus, properly balancing the exploitation of guaranteed resources and exploration of uncertain options is critical for survival and success [7]. The exploration–exploitation trade-off is the basic rule of decision-making that people are confronted with when they have to decide between staying with proven exploitative opportunities and exploring new untapped ones with the promise of higher returns in the future. Exploration is seeking new knowledge or experimenting with new behavior at the cost of less short-term reward, and exploitation is applying learned knowledge or proven behavior to achieve maximum current rewards. This balance is a fundamental element of a wide range of decision-making problems, from animal foraging to human learning and artificial intelligence such as reinforcement learning. Closely related is stay-leave behavior, which is the selection of staying in a current option or leaving it for another, typically observed in contexts such as foraging or multi-armed bandit tasks. The stay-or-leave decision is based on factors such as expected reward, risk aversion, and uncertainty, and it indicates the ability to calculate the opportunity cost of resources and time. In the context of neuropsychiatric illness, decision-making impairments can lead to cognitive rigidity, which can promote maladaptive behavior such as compulsive gambling, drug use disorders, or failure to adapt in dynamic situations. These decision-making systems are influenced by neural activity, particularly in areas that are involved in reward processing and cognitive control, including the dopamine system. Understanding the exploration–exploitation trade-off dynamics and stay-leave behavior is crucial to explore the cognitive and neural bases of heterogeneous psychiatric and neurological disorders [11, 23, 24, 27, 76]. Psychiatric disorders have been increasingly investigated for potential impacts on this delicate exploration–exploitation equilibrium, manifesting as deficits in flexibly switching between choices and optimizing behavior. Many studies have employed specialized decision-making tasks like n-armed bandit [5, 27, 37, 41, 42], foraging [24, 49, 53, 54, 70], reversal learning, and Iowa Gambling [12,13,14] to probe these facets across various mental disorders [18, 36, 71, 82]. Various disorders significantly impact cognitive processes, manifesting through distinct but sometimes overlapping patterns of impairment. In addiction-related disorders, such as pathological gambling (PG), alcohol dependence, and cocaine use, individuals typically exhibit deficits in decision-making, feedback processing, and the ability to make advantageous choices [25, 39]. These impairments often stem from difficulties in focusing attention, switching between cognitive patterns, and controlling impulsive behaviors [65]. Neurodegenerative disorders, including Parkinson's disease (PD) and neurodevelopmental disorders, including Autism Spectrum Disorder (ASD), present unique cognitive challenges. PD patients experience selective deficits in reward processing and novelty seeking, which are modulated by dopaminergic drugs. These medications can enhance reward processing but often disrupt punishment processing, indicating a complex interplay between neurochemical treatment and cognitive function [17, 70]. Individuals with ASD show distinctive decision-making patterns, characterized by slower learning from positive feedback and a tendency to constantly shift between choices, indicating a broader issue with integrating feedback over time [63, 89]. In the realm of mood and psychiatric disorders, conditions such as depression, anxiety, and schizophrenia (SZ) further illustrate the diverse impact on cognitive processes. Depression is associated with impaired decision-making, particularly in dynamic environments, due to altered sensitivity to reward and punishment [16, 20]. High trait anxiety leads to an attentional bias towards aversive stimuli, affecting overall cognitive functioning [28]. Schizophrenia patients exhibit reinforcement learning abnormalities, particularly in their ability to integrate positive decision outcomes over time [77, 78],[85]. ADHD, meanwhile, is characterized by deficiencies in reward prediction and decision-making, often linked to impaired learning processes [45]. An emerging field of computational psychiatry seeks to understand underlying computations in the brain and examines what components are altered in psychiatric disorders. One of the main components in decision-making and learning is exploration–exploitation balance.

The present systematic review aimed to elucidate patterns of exploration–exploitation impairments across psychiatric disorders, examining:

• 1. which precise capacities were disrupted in each illness?

• 2. Similarities and differences between disorders.

• 3. How diseases affected decision-making balancing.

Clarifying these effects will advance our comprehension of the neurocognitive bases of adaptive choice behavior. Elucidating common and distinct influences on the dynamics of exploring uncertain options versus exploiting known rewards provides fundamental insights into underlying decision processes. This knowledge can inform interventions to restore the delicate yet vital equilibrium between exploitation and exploration.

Method

Search strategy

Articles up to April 12, 2023, were included in the search, an electronic database (PubMed) without limits on the year of publication was searched for relevant journal articles. The database was queried with keywords including one of these three dimensions: 1) foraging task, Patch leaving, and opportunity cost; 2) explore-exploit and Bandit task; or 3) Mental Disorders and Psychiatry Disorders.

The keywords, with the following syntax:

(forag* OR"patch leaving"OR patch-leav* OR"Explore/Exploit"OR"explore-exploit"OR"bandit"OR"opportunity cost"OR"opportunity-cost"OR"exploration–exploitation"OR"exploration exploitation"OR"Reinforcement learning"OR"exploratory behavior"OR"IGT"OR"Iowa gambling"OR"Reversal learning") AND ("Mental Disorders"[Mesh] OR"mental"OR Tourette OR dyslexia OR psychiatr* OR"Anxiety"OR"Panic"OR MDD OR BMD OR"major depression"OR"major depressive"OR"Alzheimer"OR Parkinson OR"brain injury"OR stroke OR TBI OR"minor depression"OR"minor depressive"OR"Bipolar"OR Gambl* OR"Dissociative"OR"Eating Disorders"OR"Anorexia Nervosa"OR"Binge-Eating"OR"Depressive Disorder"OR"Amnesia"OR"Dementia"OR schizo* OR"ADHD"OR"OCD"OR"obsessive"OR"compulsive"OR"attention-deficit"OR"attention deficit"OR"hyper-activity"OR"hyperactivity"OR Addic* OR"Tic Disorder"OR"PTSD"OR"Post-traumatic stress"OR"Learning Disability"OR"Learning Disabilities"OR"Sadism"OR autis* OR abuse OR"personality disorder"OR borderline OR sleep OR phobia OR alcohol OR substanc* OR smoke OR cigarette OR"dependent personality"OR attachment).

Eligibility criteria

Selected studies had to fulfill the following inclusion criteria:

• 1. The study included human subjects.

• 2. The publications were in English and peer-reviewed.

• 3. Full-text availability.

• 4. The study included at least one group with a psychiatric disorder and a healthy control group.

• 5. Studies had to incorporate a task or paradigm to assess an aspect of the explore-exploit dilemma, foraging task, Patch leaving, or stay/switch.

In this systematic review, the first two authors conducted the study selection process independently. Their screening of titles and abstracts for entry was based on a pre-defined set of inclusion criteria. Disagreements and uncertainties were solved through discussion while the third reviewer, Author Third, was involved in resolving persistent disputes. This ensured that only appropriate studies were included for full-text review fairly and independently.

The reference lists of all included articles, including those that did not meet our inclusion criteria, were manually screened to identify additional eligible primary studies that may have been missed in the initial database search.

The complete list of articles was exported to EndNote X7 and was subsequently for the title and abstract screening. The titles and abstracts obtained with the search criteria in the database were independently screened. Then, each full text was screened for inclusion criteria. Reasons for exclusion were recorded. Additional articles were then added when appropriate.

The inclusion criteria required studies with both a clinical population and an experimental task that included aspects of the exploration–exploitation trade-off. We used fairly flexible inclusion and exclusion criteria, keeping several studies that looked at disorder-related traits such as anxiety, stress, and externalizing behaviors, rather than diagnosed disorders themselves. These additional studies, shown in Table 1, were kept because they offered useful insights relevant to this review.

Table 1 Characteristics and main results of studies included in the systematic review

Results

Study identification and selection

A total of 17,878 unique abstracts were included in this review (Fig. 1). After screening titles and abstracts, 264 articles were identified as potentially relevant. The full texts of these articles were reviewed to determine inclusion or exclusion, and 13 additional records were identified through other sources. Of the 277 articles that underwent full-text screening, 231 studies were excluded for the following reasons: not being human studies, lacking appropriate tasks, having healthy volunteers or no patient groups, not examining exploration–exploitation or foraging concepts, or being inconsistent with the study aim. Ultimately, 46 studies met all inclusion criteria and were selected. It is worth noting that a considerable number of articles on reinforcement learning were not included here, either because they did not directly address the topic of interest or only tangentially examined it.

Fig. 1

Study Selection Process

Study characteristics

The characteristics of each included study are presented in Table 1. A total of 2468 patients were compared to 1881 healthy controls across the 46 studies. The inclusion criteria were as follows: 15 studies utilized the n-armed bandit task, 5 studies used a foraging task paradigm, 14 studies employed probabilistic reversal learning (PRL), 11 studies used the Iowa Gambling Task (IGT), one study used a choice task, one study included a risk task and one study used a continuous performance Temporal Utility Integration Task. Notably, over 456 articles examined PRL and IGT, but mostly those assessing exploration–exploitation dilemmas, patch leaving/foraging concepts, or stay/switch behavior were eligible for inclusion. Thus, a limited number of PRL and IGT studies met the criteria.

Outcome measures and results

In this study, three categories of behavioral outcome measures were examined to assess the exploration–exploitation trade-off:

• 1. Stay-Leave or Stay-Switch Behavior: This behavior was assessed using the foraging task paradigm. In this task, participants had to decide whether to stay in an existing resource or search for new resources.

• 2. Exploration–Exploitation Patterns: These patterns were evaluated using the n-armed bandit task. In this task, participants had to choose between different options with probabilistic rewards and maintain a balance between exploring new options and exploiting known options.

• 3. Shifting and Switching: Assessment of shifting and switching behaviors was conducted through the IGT and PRL. In these tasks, participants had to be able to timely change their strategies and switch to new options.

This categorization was made due to the differences in the nature and characteristics of each of these tasks. Each task examined different aspects of the exploration–exploitation trade-off. The foraging task focused on the decision to stay or leave a resource. The n-armed bandit task focused on choosing between different options with probabilistic rewards. The IGT and probabilistic reversal learning assessed the ability to switch and change strategies.

It is worth noting that the stay-leave or switch behavior may have also been examined in other studies using reinforcement learning approaches, but in this systematic review, the stay-leave behavior was considered only from the perspective of the foraging task paradigm. The 4-armed bandit task and the patch-leaving task are both experimental paradigms used in psychology and behavioral economics to study decision-making, reinforcement learning, and exploration–exploitation trade-offs.

While they share some similarities, they have distinct characteristics: The n-armed bandit task typically involves discrete choices between a small number of options with probabilistic rewards, while the patch-leaving task involves continuous decision-making in spatial environments with uncertain resource distributions.

While both tasks involve exploring and exploiting options, the nature of exploration and exploitation differs. In the n-armed bandit task, exploration involves trying out different options to learn their reward probabilities, while in the patch-leaving task, exploration involves searching for new resource patches with potentially higher rates. The patch-leaving task often involves spatial navigation and memory processes, as participants must remember the locations and qualities of different patches. This aspect is less prominent in the n-armed bandit task.

We used measures such as stay-leave, win-stay, lose-shift, choice consistency, inverse temperature of softmax in models, switch after reversal, later switch in IGT after large punishments, model-based dissociation between random exploration and directed exploration and similar measures as an indicator of exploration–exploitation trade-off. In some studies, the results were so clear and in some studies, we inferred from results and switching patterns that were reported.

In summary, while all three tasks study decision-making under uncertainty, they differ in their specific setups, feedback mechanisms, temporal dynamics, and cognitive processes involved. The 4-armed bandit task focuses on discrete choice with immediate feedback, the patch-leaving task involves spatial navigation and resource management, and the IGTexamines decision-making in a risk-reward context with abstract options.

Tasks

Foraging task

In foraging tasks, individuals collect rewards by searching patches that vary in their available number of rewards (Fig. 2) [15, 88]. People can make decisions based on the trade-off between exploiting a particular patch and exploring to find a new patch with potentially higher available rewards. For instance, consider the scenario of berry collection, where an individual first harvests a specific blackberry plant. As time passes, most of the blackberries on that particular plant will be picked, making it increasingly time-consuming to find the remaining ones. At this stage, the picker faces a choice: continue searching for blackberries on the same plant or venture elsewhere, even if it means spending more time on travel and uncertainty regarding the quality of the next plant. The exploration–exploitation dilemma in foraging has been extensively studied in various animal species [22, 46].

Fig. 2

The foraging task involves two visual layers. The aerial view displays a game environment with a movable character aiming to find hidden resource patches. Upon approach, patches become partly visible. The patch view provides a zoomed-in perspective after entering a patch. Resources (berries) are randomly distributed within patches. The goal is to collect as many berries as possible within a time limit. This two-layered structure examines how individuals balance patch exploitation with aerial exploration to optimize resource collection [81]

The bandit task

Another task widely used for probing how people trade off exploration and exploitation is the—n-armed bandit task, here, participants are asked to make repeated choices between multiple-choice options (“bandits”) to acquire rewards. Exploitation involves chasing each bandit’s expected value and selecting the best. In contrast, exploration can be undirected because of the stochastic selection of bandits (“random exploration”) [23, 27, 84]. Two important methods for exploration–exploitation are epsilon-greedy and softmax methods: in the first one in the epsilon fraction of trials, a non-best option is chosen randomly. In the softmax method, options are chosen with the probability that are in line with their value after passing through a Boltzmann (softmax) function.

To illustrate, consider a"four-armed bandit"task, which consists of repeated choices among four slot machines (Fig. 3). These slot machines provide payoffs in the form of points that can be exchanged for money. Importantly, the payoffs of the slot machines fluctuate noisily around four different means. Unlike traditional slot machines, the mean payoffs of these bandit machines change randomly and independently from trial to trial. Participants actively sample these machines to gather information about the current value of each slot. This experimental design, coupled with model-based analysis, allows researchers to study both exploratory and exploitative decisions within a single task [27].

Fig. 3

In the experimental"four-armed bandit task,"participants repeatedly select one of four slot machines. Each slot machine awards points that can be converted into cash, and the average payouts of the machines vary at random and independently from one trial to the next. To learn about their current value and make choices based on their perceptions of which slot machine offers the best payouts, participants must actively sample the machines. The goal of the task is to investigate exploratory and exploitative decision-making under ambiguous circumstances [27]

Probabilistic reinforcement learning (PRL)

Probabilistic reversal learning (PRL) serves as a robust behavioral task employed to evaluate the delicate balance between factors such as cognitive flexibility, impulsivity, and compulsivity. This task enables the investigation of the impact of positive or negative feedback on learning within various neurological, neurodevelopmental and psychiatric conditions, including Huntington's disease, ASD and schizophrenia, in both patient populations [51, 61].

The Iowa Gambling Task (IGT)

The task involves four decks of cards, and participants are asked to draw cards from these decks, to maximize their winnings over a series of trials. Some decks are advantageous in the long run (leading to overall positive outcomes but occasional losses), while others are disadvantageous (leading to immediate gains but long-term losses). The challenge for participants is to figure out which decks are more beneficial and to adjust their choices accordingly [12].

Findings

We categorized the disorders observed in the reviewed studies into three proposed groups to aid in better conceptual organization: A) Addiction-Related and Compulsive Conditions, B) Mood and Psychiatric Conditions, C) Neurological (Neurodevelopmental and neurodegenerative) Conditions.

Addiction-related and compulsive conditions

A category encloses the disorders characterized by dysregulated reward circuits and impaired behavioral inhibition, where decision-making processes are fundamentally altered by maladaptive reward sensitivity and compulsive behavioral patterns. Gambling disorder (GD) is an addiction characterized by difficulties in decision-making based on value and behavioral adaptability.

Wiehler and colleagues performed a four-armed bandit task during functional magnetic resonance imaging (fMRI) to quantify and compare exploration behavior between the two groups. Participants’ choices were analyzed using computational modeling with reinforcement learning algorithms of varying complexity. The computational modeling demonstrated reduced directed exploration; directed exploration is dissociated from random exploration in this study, and hints towards an exploration that purposefully search for new information and choose in way that improve the information content. However, fMRI analysis showed no significant differences in brain activity between gamblers and controls, except for decreased activation during direct exploration specifically in the substantia nigra/ventral tegmental area which are dopaminergic areas. So this can relate the directed exploration to dopaminergic activity in the brain [87].

In contrast, Addicott and colleagues hypothesized that gambling may utilize similar reward circuitry in the brain as addictive substances. To test this, they employed four-armed bandit and patch foraging tasks. Their findings revealed a positive correlation between gambling frequency/beliefs and exploratory decision-making. Individuals who gambled weekly exhibited poorer performance compared to those who gambled yearly on both tasks. Additionally, stronger gambling perceptions were associated with prematurely exiting patch foraging. Overall, the authors conclude frequent gamblers show suboptimal abilities to maximize rewards, suggesting they may ineffectively search and forage natural resources.

Meanwhile, using a probabilistic reinforcement learning paradigm, Perandrés-Gómez, and colleagues discovered that individuals with gambling though were prone to shifting choices after single instances of reward or punishment, yet also persisted in the same decision despite multiple consecutive penalties. This implies a greater tendency to persevere with choices in the face of accumulated negative feedback [66]. This finding is consistent with [87] that show a reduction in directed exploration.

In another study, Abram et al. found that attractive rewards or incentives related to taking high risks and potentially incurring losses can reliably predict an individual's subsequent decision to pursue additional rewards. In other words, the choices people make after being enticed by high-risk but potentially rewarding situations can be anticipated based on the initial appealing incentives. Conversely, in the absence of such losses, there appears to be no discernible impact on the processing of reward value or the pursuit of rewards. Notably, individuals with higher externalizing traits that are at a higher risk of developing addiction, exhibited no decline in their propensity to select subsequent risky options. These findings strongly imply that the inability to learn from errors may represent a significant contributing factor to the risk of addiction [3].

Furthermore, Morris et al., employing a clock arm rotating at 5 s per revolution, found that patients diagnosed with Alcohol Use Disorder (AUD) exhibited reduced exploratory behavior in both gain and loss domains, leading to a decrease in the efficiency of exploitative choices. Interestingly, obese subjects with and without Binge Eating Disorder (BED) were not significantly different. However, when compared to each other or AUD participants, individuals with BED displayed heightened exploratory behaviors, particularly in the loss domain [62]. Lastly, Goudriaan et al. investigated decision-making deficits in pathological gambling (PG) and alcohol dependence (AD) groups, comparing them to control. Their findings indicated that both the PG and AD groups exhibited deficits in decision-making processes, attributed to deficiencies in feedback processing after losses. Interestingly, this pattern did not extend to the Tourette syndrome (TS) group. Furthermore, subgroup analyses revealed larger decision-making deficits in pathological slot machine gamblers than in pathological casino gamblers. Additionally, the results indicated that both the PG and AD groups selected fewer advantageous decks than the control group, with only the PG group exhibiting faster decision-making than the control group and deficits in decision-making strategies, persistence in seeking rewards, and shortcomings in learning discrimination with reward and loss scenarios each have unique yet cumulative impacts on the emergence and persistence of gambling issues. Subsequent analyses indicated that the healthy control group shifted more after losses than after rewards, while this effect was almost absent in the PG group. In conclusion, PG individuals show an exploitative bias, persisting in suboptimal choices due to poor feedback adaptation [39].

Individuals with substance use disorders experience a disrupted explore-exploit balance due to impaired learning and feedback processing. Smith et al., through the application of computational modeling, multiple regression, and hierarchical Bayesian group analyses, delved into the investigation of how individuals with Substance Use Disorders (SUDs) compared to controls in resolving uncertainty within a gambling task. While SUD participants won less frequently than controls most of the time, both groups exhibited similar reward sensitivity and insensitivity to data, influencing their exploration strategies. However, SUD participants showed distinct characteristics, including low action precision, high learning rates for rewards, and low learning rates overall. Consequently, this suggests that poor performance in explore/exploit decisions in SUDs arises not only from incompatible selections, especially in the face of positive outcomes, but also from suboptimal learning rates for rewarding vs. non-rewarding outcomes [74].

In another study, Harle et al., employing a two-armed bandit task, embarked on a comparison of learning and decision-making in individuals addicted to methamphetamine (MDI) with controls. Their findings highlighted significant differences between HV and MDI individuals in their rate of reward learning and the subsequent use of that information to make decisions. HV participants demonstrated the tendency to learn from feedback but gave disproportionate weight to recent data. Consequently, their decision strategies were best fit as proposed version of Softmax model. Conversely, MDI individuals appeared to pursue an ordinary and independent learning policy, akin to a Win-stay/Lose-shift (WSLS) strategy. Despite this, their choices had a low probability of aligning with the optimal option that provided the most reward [42].

Additionally, Addicott et al.'s investigating explore/exploit behavior in cigarette smokers and its neural correlates emphasized the cognitive effort required for exploratory decision-making. Their results indicated that increased self-reported smoking behavior correlated with enhanced brain activation during exploratory decision-making, suggesting that exploratory decision-making necessitates more cognitive effort [4]. Moreover, Addicott et al. revealed that smokers engaged in a bandit task exhibited lower exploratory behavior than non-smokers and made fewer switches between bandit arms [8].

Patzelt, E. H. et al., utilizing a reversal-learning task, illuminated that individuals addicted to cocaine displayed more frequent switching behavior in response to false feedback and spontaneous shifts in the reversal-learning task [65].

Robinson et al. displayed that people with methamphetamine use disorder (PwMUD) experience a breakdown in the exploration–exploitation dilemma due to deficits in reinforcement learning rather than inflexibility. Their difficulty in capitalizing on stable reward contingencies stems from poor learning performance and weak action-reward associations. Instead of committing to beneficial choices, PwMUD exhibits excessive exploration, frequently switching responses based on recent feedback rather than relying on accumulated experience. However, the lack of heightened perseveration suggests that these deficits arise from unstable learning processes rather than compulsive behavior. These findings indicate that treatment strategies should focus on enhancing reinforcement learning processes rather than solely targeting compulsivity [68].

Cordovil et al. investigated the impact of alcohol withdrawal on affective selective attention and executive functions (EFs) in alcohol-dependent patients. In comparison to control subjects, alcohol-dependent patients exhibited reduced abilities to focus on relevant information and poor switching performance in making advantageous choices [25].

Collectively, these studies highlighted that Individuals with substance use disorders display varied explore-exploit behaviors, with some showing excessive exploration due to unstable learning (e.g., methamphetamine users) and others favoring rigid exploitation (e.g., smokers). These deficits stem from impaired reinforcement learning rather than pure compulsivity, highlighting the need for treatment strategies that enhance adaptive decision-making.

Eating disorders, like other compulsive behaviors and addictions, disrupt decision-making by altering the balance between exploring new options and rigidly sticking to familiar choices. Kristjánsson and colleagues utilized a foraging task paradigm to reveal differences in switch costs between an eating disorder symptom group and a control group. The symptom group exhibited higher switch costs [49].

Supporting this, Verharen and colleagues computationally modeled data from anorexia nervosa patients and controls performing the IGTbased on prospect utility theory tenets. Compared to controls, anorexia nervosa patients showed intact reward sensitivity, value learning, and exploration–exploitation balances. However, they displayed reduced loss aversion typically seen in healthy populations [82].

Giannunzio and colleagues investigated decision-making in a substantial sample of adolescent and adult anorexia nervosa patients. They analyzed data using two models that assess exploratory abilities/reward-punishment sensitivity versus learning from feedback to optimize long-term gains once rules are inferred. On average, adult anorexia nervosa patients exhibited poorer decision-making, with deficits in exploration and reversal learning, alongside a bias toward disadvantageous choices. Both age groups showed heightened loss avoidance compared to controls [36]. To sum up, Individuals with eating disorders exhibit impaired explore-exploit trade-offs, with anorexia nervosa patients showing deficits in exploration, heightened loss avoidance, and a bias toward disadvantageous choices. These decision-making impairments highlight the need for interventions that enhance cognitive flexibility and adaptive learning.

Mood and psychiatric disorders

In this section we investigate the mood and psychiatric disorders that fundamentally disrupt cognitive adaptability. We reveal how emotional and cognitive variations modulate exploration–exploitation dynamics through altered uncertainty perception and adaptive learning mechanisms. Anxiety, like other mood disorders, disrupts decision-making by altering how individuals balance exploring new possibilities and committing to familiar choices, often leading to maladaptive behaviors.

Marzuki and colleagues utilized a PRL task to reveal increased reward pursuit and exploratory choices in young OCD patients regardless of medication status, alongside decreased sensitivity to punishments [58].

Similar to Marzuki’s findings, Aberg and colleagues employed a multi-armed bandit task, discovering increased exploratory choices in those with higher trait anxiety, regardless of gain or loss contexts. They also found anxiety reduces exploratory drives arising from uncertainty [1].

In line with these findings, Aylward and colleagues computationally investigated learning under uncertainty in both healthy and mood/anxiety disorder individuals. Their findings suggested high anxiety and mood symptoms shift the exploration–exploitation balance toward heightened exploration [10].

In their findings, Visser and colleagues linked trait anxiety to gender differences in complex decision-making, assessing healthy volunteers with the Iowa Gambling Task. Anxiety disrupted men's early exploratory performance but impaired women in later exploitative stages [28].

Finally, Lamba and colleagues hypothesized anxious individuals may have more difficulty learning in uncertain social settings. Their bandit task revealed control group subjects swiftly identified when to cease investing in exploitative partners under social uncertainty. However, anxiety led to excessive investment due to discounting learning from negative social outcomes [50]. Anxiety generally shifts the explore-exploit trade-off toward increased exploration, with individuals displaying heightened exploratory choices and difficulty learning from negative outcomes. However, uncertainty can drive more exploitative behavior, suggesting that anxiety’s impact on decision-making depends on context and individual differences. While anxiety pushes individuals toward constant exploration, depression often locks them into a cycle of indecision, reducing the drive to explore new opportunities and amplifying the tendency to avoid change. Harle and colleagues investigated how variations in affective measurements relate to different facets of reward-based learning and decision-making in depression. Utilizing a two-armed bandit task, they discovered increased stochasticity and exploratory tendencies in depressive individuals with high anhedonia. It appears anhedonia is associated with reduced valuation of instant rewards. Additionally, the researchers found depressive individuals with anhedonia were less inclined toward reward maximization, while anxious participants displayed greater reliance on independent win-stay/lose-shift learning strategies [41].

In a comparable finding, Cella and colleagues assessed and compared flexibility in decision-making between medicated unipolar major depressive disorder patients and controls using the Iowa Gambling Task. Their findings revealed deficits in depressive patients during both standard and contingency-shift phases, implying greater dependence on previous positive emotional outcomes and reduced adaptability in transition toward new advantageous behaviors [20].

In contrast, Dombrovski and colleagues examined reversal learning in elderly individuals with a history of suicide attempts and depression. Their results indicated no significant differences emerged in reward learning rates or exploratory drive between these depressed participants and controls [30]. In conclusion, individuals with depression, particularly those with high anhedonia, show increased exploration but reduced reward maximization and adaptability, highlighting difficulties in adjusting decision-making strategies and learning from past outcomes.

Additionally, Urošević and colleagues studied reward and threat sensitivity correlates of probabilistic decision-making in teenagers with and without bipolar disorder. Relative to controls, teens with bipolar disorder displayed a lower tendency to persist with prior positively reinforced options. Comparison between groups suggested enhancing win-stay rates constitutes a more optimal learning approach [80].

While anxiety and depression disrupt decision-making by skewing the explore-exploit balance, schizophrenia exacerbates this imbalance, causing unpredictable shifts and an inability to consistently leverage past rewards. Saperia and colleagues utilized the IGT to examine win-stay/lose-shift behavior and evaluate the effects of rewards and penalties guiding real-world decisions in schizophrenia and depression. They revealed unaltered lose-shift yet reduced win-stay rates in schizophrenia patients compared to controls, significantly linked to motivational and cognitive deficits. In contrast, no differences emerged in depressed patients [71]. In agreement with Saperia’s results, Culbreth and colleagues uncovered inefficient win-stay/lose-shift strategies and reduced feedback learning in schizophrenia patients using a PRL task [26]. Moreover, Cathomas and colleagues found performance reductions in schizophrenia patients on a bandit task attributable to over-switching from exploitation to random exploration relative to controls [19].

Similarly, Martinelli and colleagues revealed schizophrenia patients were attracted to new options over learned high-value choices in a multi-armed bandit task, potentially disrupting balanced exploration–exploitation tradeoffs [57]. Additionally, Reddy and colleagues identified more frequent switching in response to both positive and negative feedback among schizophrenia outpatients compared to control subjects on a PRL task, reflecting indiscriminate shifting [67].

Waltz and colleagues also demonstrated early learning stage instability in schizophrenia patients, with excessive switching regardless of feedback on a similar PRL task [86]. Furthermore, Matsuzawa and colleagues revealed failures to shift from disadvantageous to beneficial decks in schizophrenia patients despite large penalties on the Iowa Gambling Task [60].

Moreover, Strauss and colleagues found schizophrenia (SZ) patients were reluctant to explore potentially superior choices. SZ patients demonstrated a notable and consistent decrease in their inclination to make exploratory behavioral changes toward responses that might produce greater expected values compared to those achieved by maintaining the current situation [77, 78]

Finally, Vinckier and colleagues mimicked the psychosis with low dose ketamine and proposed uncertainty in psychosis explains failures to leverage environmental patterns, consistent with evidence showing poor reversal learning. They investigated confidence-driven transition towards the exploitation of learned contingencies and showed that ketamine as a proxy to psychosis reduces this confidence dependent weight and so increase the exploration [83].

In summary, Schizophrenia patients exhibit significant deficits in decision-making, including excessive switching between options and a reduced ability to exploit learned choices, reflecting both cognitive and motivational challenges. These patterns of erratic exploration and poor feedback learning highlight the disruption of the explore-exploit trade-off in schizophrenia, distinguishing it from other mood disorders.

Neurological (Neurodevelopmental and Neurodegenerative) Disorders

Investigation of neurologically-rooted disorders that systematically impair social learning, adaptive decision-making, and behavioral regulation, highlighting neurocognitive mechanisms underlying exploration–exploitation trade-offs. ASD is a neurodevelopmental condition marked by ongoing difficulties in social communication and interaction, along with the existence of limited, repetitive behaviors, interests, or activities.

Addicott and their research team investigated the impact of ADHD status and the administration of methylphenidate, a common medication for ADHD, on the decision-making process involving exploration and exploitation. This was achieved through the utilization of a 6-armed bandit task. Their results showed that people with ADHD tended to explore more, but their exploratory behavior was suboptimal. Notably, they displayed a recurring pattern of selecting low-value options, even at the expense of optimizing their rewards, ultimately resulting in fewer overall gains [6].

In a similar vein, Hauser and their colleagues reported a phenomenon of amplified yet relatively simplistic exploratory behaviors among individuals diagnosed with ADHD when compared to a control group in a probabilistic reinforcement learning task [44].

In another study, Hauser and colleagues shed light on the heightened variability observed within the ADHD population, which could be indicative of an altered balance between exploration and exploitation. Their use of a reversal-learning task brought to light that only a minority of individuals with ADHD demonstrated the ability to effectively acquire knowledge through exploratory approaches [45]. In sum can be concluded that Individuals with ADHD exhibit a tendency to engage in excessive and often ineffective exploration, selecting low-value options and demonstrating high variability in their decision-making. This disrupted balance between exploration and exploitation highlights challenges in optimizing rewards, underscoring the need for strategies to improve decision-making efficiency.

Individuals with ASD face distinct challenges in decision-making, often displaying rigid behaviors that impact their ability to balance exploration and exploitation. Carlisi and colleagues conducted a comparative analysis of decision-making in individuals with ASD and Obsessive–Compulsive Disorder (OCD). Both ASD and OCD groups displayed a reduction in choice consistency and lower reinforcement learning abilities compared to the control group. Additionally, individuals with ASD showed specific abnormalities in choice perseverance, and ASD adolescents exhibited increased switching behavior. Interestingly, OCD patients demonstrated similar heightened exploration tendencies to those observed in individuals with ASD, regardless of outcome sensitivity [18].

Building upon this, Mussey and their team utilized the IGTand reported that patients with ASD, in contrast to the control group, exhibited a higher frequency of deck-switching, which led to suboptimal decision-making. Furthermore, these patients displayed a slower learning curve in identifying advantageous decks, aligning with the notion of reduced reinforcement learning in ASD individuals [63].

In contrast to these patterns, Yechiam et al. utilized the IGT to reveal a unique adaptive learning style among certain individuals with ASD. This style was characterized by reduced sensitivity to immediate motivational structures and an intensive exploratory search for available alternatives. It's noteworthy that this distinct learning approach, advantageous in specific contexts, may be perceived as abnormal in social settings [89].

Finally, Solomon et al., using Probabilistic Reinforcement Learning, delved into the performance of adults with Autism ASD. Their research corroborated earlier findings, indicating that individuals with ASD encounter challenges in utilizing positive feedback to exploit reward choices, resulting in a slower learning rate. This pattern reinforces the overarching theme observed in individuals with ASD across various decision-making tasks [75].

It could concluded that for individuals with ASD, the balance between exploration and exploitation seems to be affected, frequently favoring increased exploration or diminished exploitation. This disparity is shown in actions like heightened switching among choices, diminished responsiveness to rewards, and difficulties in using feedback to enhance decision-making. These patterns indicate that people with ASD might focus on acquiring information rather than optimizing known rewards, possibly detracting from effective learning and adaptive results.

Parkinson’s Disease (PD) is a neurodegenerative disorder that caused by deficiency in dopaminergic pathway. Studies that investigate the exploration–exploitation balance in this disease can reveal the role of dopaminergic system in exploration. Constantino et al. investigated the role of tonic dopamine (DA) replacement medication in PD patients, focusing on both ON and OFF medication states using the patch-foraging task. They aimed to understand the influence of dopaminergic opportunity cost signals on successive decision-making. Their results revealed that the absence of DA reduced the mental opportunity cost of time. In contrast, PD patients in the OFF medication state exhibited a lower threshold for rewards compared to the control group. However, the administration of tonic dopamine DA substitution medication can make a relief on this deficit.

Expanding upon this research, Rutledge et al. [70] explored the impact of dopaminergic drugs on learning rates and perseveration in PD patients, employing a foraging task. Their investigation found that dopaminergic medication increased learning rates in PD patients. Interestingly, learning rates in PD patients off dopamine medication resembled those of the aged control group. Furthermore, a comparative analysis of learning rates in PD patients with individuals experiencing milder disease progression or the elderly demonstrated that PD patients exhibited diminished learning rates. Notably, the medication appeared to selectively enhance learning rates for positive outcomes while having no such effect on negative outcomes.

Bodi et al. [17] studied the effect of dopamine agonists on reward and punishment processing in never-medicated, young Parkinson's disease (PD) patients. They demonstrated that never-medicated PD patients had decreased novelty-seeking and reward processing. Dopamine agonists (pramipexole and ropinirole) ameliorated novelty seeking and reward processing, augmenting the interaction between both behaviors and interfering with punishment learning without influencing harm avoidance. Most importantly, the study highlighted that PD patients, due to dopaminergic dysfunction, have difficulty in achieving a balance between exploration–exploitation trade-offs. In the unmedicated state, they were biased toward exploration (novelty seeking) but lacked efficiency in reward processing to enact exploitation (exploiting current rewards). Dopamine agonists corrected the imbalance by improving reward-based learning and thus facilitating a more effective exploration–exploitation strategy.

In a distinct context, Djamshidian et al. [29] assessed novelty-seeking behavior in PD patients using a three-armed bandit task. They categorized patients into non-impulsive and impulsive-compulsive behavior (ICB) groups and assessed their behavior in both ON and OFF dopaminergic medication states, comparing it to a control group. The findings revealed that PD patients with ICB displayed a significantly stronger inclination toward choosing novel options, irrespective of medication status, in contrast to non-impulsive PD patients or controls.

In another related study focusing on reward and punishment processing in PD patients, Bo´di et al. [17] employed a probabilistic classification task. Their investigation centered on young, never-medicated PD patients and identified a significant reduction in novelty-seeking and reward processing, both of which were interrelated.

In conclusion, Parkinson's disease disrupts the balance between exploration and exploitation due to dopaminergic dysfunction, leading to excessive exploration and impaired reward processing. However, dopaminergic treatment can alleviate these deficits, enhancing learning rates and improving the ability to balance exploration and exploitation effectively.

Table 2 illustrates the impact of various disorders on decision-making, particularly regarding whether people are inclined to prefer exploitation (remaining with familiar rewards) or exploration (pursuing new alternatives). The Cohen’s d effect sizes reveal the intensity and orientation of these patterns when data is present. Although the variety of measures and tasks employed in different studies varies, we endeavored to standardize the results by computing effect sizes whenever feasible. In Table 2, we display the Cohen’s d values along with the direction of the noted differences between individuals having different disorders and the control groups. This method facilitates relative comparisons among disorders, aiding in the recognition of wider patterns in decision-making habits. Based on the gathered information, we notice a significant trend in specific conditions. ADHD, Autism Spectrum Disorder (ASD), and Schizophrenia show a higher inclination for over-exploration, indicating that people with these conditions tend to pursue new alternatives instead of remaining with familiar options. This behavior may be associated with problems in cognitive flexibility, increased sensitivity to uncertainty, or variations in dopamine regulation that impact reinforcement learning.

Table 2 Shows the frequency of study’s conclusions on the direction of exploration and exploitation in disorders with the corresponding Cohen’s d effect sizes in accolades (when information was available to calculate)

In contrast, Parkinson’s disease, pathological gambling, and specific addictions reveal a tendency for over-exploitation, as people often continue to pursue choices that were previously rewarding instead of seeking out new options. This might indicate a diminished capacity to adapt to evolving surroundings, possibly caused by deficiencies in reward learning, habit development, or compromised cognitive control functions. For anxiety, eating disorders, and certain types of addiction or substance use disorders, the results are less consistent and dependent on context. Certain research indicates a propensity for greater exploration, possibly fueled by an increased sensitivity to uncertainty or avoidance behaviors, whereas other studies reveal a greater dependence on exploitation, which may stem from inflexible thought processes or compulsive behaviors. These varied outcomes underscore the intricacy of these conditions and indicate that elements like task structure, severity of symptoms, and personal differences could greatly impact decision-making.

Discussion

This systematic review aimed to clarify how psychiatric conditions affect the intricate interplay between exploration and exploitation within decision-making processes and reinforcement learning processes. The results highlight that different mental health conditions alter this balance in distinct manners, often reflecting underlying neurocognitive and neurochemical disruptions. A useful framework organizes these conditions into three broad categories: addiction-related, emotional and cognitive, and neurodevelopmental disorders. These patterns are driven by heightened sensitivity to uncertainty, impaired reward processing, and dysfunctional cognitive control, particularly mediated by dopamine dysfunction and frontostriatal circuitry. The updated findings underscore the critical role of these mechanisms in modulating the exploration–exploitation trade-off, offering insights for targeted interventions.

• 1. Addiction-Related Conditions: Encompassing disorders defined by compulsive tendencies and reduced mastery over reward-oriented actions, this group reveals a disrupted equilibrium between exploratory and exploitative tendencies, accompanied by flawed coordination of these decision-making components.

• 2. Mood and Psychiatric Conditions: This category includes disorders distinguished by irregularities in emotional regulation, cognitive functioning, and thought processes, which shape exploration and exploitation through influences such as gender differences, immediate stress responses, and motivational triggers.

• 3. Neurological (Neurodevelopmental and Neurodegenerative) Conditions: neurodevelopmental disorders, stemming from atypical neurological maturation, these disorders are characterized by excessive, yet frequently unproductive, exploratory behavior, which impedes learning processes and the capacity to maximize beneficial outcomes. In other hand, neurodegenerative disorders such as Parkinson that reveals the role of neural structures especially dopaminergic pathways on exploratory behavior.

• 1. Addiction-Related Conditions

Disorders exhibiting addiction-like traits, coupled with weakened control over reward-focused behaviors, often present a skewed balance between exploratory and exploitative inclinations. Examples include gambling disorder, substance dependency, and behavioral compulsions. These conditions are typified by an intense focus on exploiting rewards, where persistent pursuit of gratification overshadows flexible exploration. This overexploitation is linked to a dysregulated dopaminergic system that reinforces habitual behaviors while diminishing uncertainty-driven exploration, as evidenced by impaired dopamine-related neural circuits affecting decision-making in risky contexts [4, 73, 86]. The ineffective reconciliation of exploration and exploitation in these cases is closely tied to compromised dopaminergic pathways and an amplified response to reward signals. For instance, gambling disorder exemplifies excessive exploitative tendencies fueled by flawed reward-learning mechanisms. Notably, interventions targeting neural structures such as the striatum and amygdala have demonstrated promise in recalibrating decision-making equilibrium [34, 35].

• 2. Mood and Psychiatric Conditions

Conditions characterized by disruptions in emotional stability, cognitive processes, and thought organization—such as depression, anxiety, obsessive–compulsive disorder (OCD), and schizophrenia—demonstrate a sophisticated relationship among emotional regulation, motivational influences, and decision-making dynamics. These disorders alter the balance between exploration and exploitation in subtle ways, shaped by factors including response to chronic or acute stress, gender distinctions, and sensitivity to rewards. Psychiatric disorders that increase exploration, such as anxiety disorders, enhance exploratory behavior due to an aversion to uncertainty, linked to hyperactivity in the anterior insula (aINS) and dorsal anterior cingulate cortex (dACC), which modulate exploratory decisions [2]. Depression is associated with increased exploration, as diminished reward sensitivity leads individuals to undervalue potential gains, prompting a shift toward seeking new information [16]. Similarly, ADHD promotes heightened, yet often random, exploration due to reduced dopamine transporter availability and impaired mesocorticolimbic function, resulting in impulsive, inconsistent decision-making [44]. Schizophrenia is characterized by deficits in uncertainty-driven directed exploration, linked to dopamine dysfunction and altered novelty processing in prefrontal and striatal regions [19, 57]. OCD and ASD also share increased exploratory behavior, possibly due to deficits in cognitive flexibility and reinforcement learning, contributing to excessive switching and poor adaptation to changing contingencies [18].

Conversely, some emotional and cognitive conditions increase exploitation by biasing individuals toward known choices. In schizophrenia, individuals with negative symptoms exhibit decreased exploration, potentially due to motivational deficits linked to impaired ventromedial prefrontal cortex (vmPFC) and striatal function [71]. In OCD, malfunctioning fronto-striatal networks contribute to repetitive actions and a pronounced dependence on exploitative approaches [59]. These findings highlight the diverse neural bases influencing the exploration–exploitation trade-off in this category.

• 3. Neurological (Neurodevelopmental and Neurodegenerative) Conditions

Disorders rooted in neurodevelopmental irregularities, such as ADHD and ASD, frequently arise from dysfunctions in dopaminergic, reduced dopamine transporter availability and impaired mesocorticolimbic function. These conditions are distinguished by an abundance of exploratory activity that often proves inefficient, obstructing effective learning and resulting in impulsive, inconsistent decision-making [44]. In ADHD, for example, individuals exhibit overly active exploration, which manifests as impulsiveness and challenges in maintaining focus on tasks requiring sustained exploitation. Conversely, ASD may be characterized by repetitive patterns or narrowly defined interests that restrict the scope of flexible exploration. Age plays a modulating role in these patterns: younger individuals with neurodevelopmental conditions often display pronounced novelty-seeking tendencies linked to ongoing brain maturation. ASD) increased exploratory behavior, possibly due to deficits in cognitive flexibility and reinforcement learning, contributing to excessive switching and poor adaptation to changing contingencies [18]. From perspective of age, it is interesting that older adults experiencing cognitive decline may lean heavily on exploitative strategies [55]. This contrast hint toward an important confounding factor, age, on the findings. Though most studies have age-matched control group, but always the caution on age for any conclusion should be taken [see theoretical integration section below].

In neurodegenerative disorders, PD is mostly investigated and considering the low dopaminergic activity, patients show more exploitative behavior and reduced novelty seeking, while dopaminergic treatment compensate this deficit. This shows an overemphasized role of dopamine in exploratory behavior.

Systematic task comparisons and goals: conceptual framework

Central to the exploration–exploitation dilemma is the inherent conflict between pursuing new possibilities and capitalizing on established rewards—a delicate equilibrium frequently disturbed in psychiatric conditions. Experimental approaches, such as bandit tasks, reversal learning exercises, and foraging scenarios, provide effective means to investigate these dynamics, particularly when paired with computational frameworks like Markov decision processes (MDPs). Future studies could leverage advanced computational tools such as Bayesian hierarchical modeling and deep reinforcement learning to capture individual variability in exploration–exploitation patterns, offering greater precision over traditional reinforcement learning models [9, 91].

These models illuminate how rising uncertainty tends to steer behavior toward exploration, whereas shrinking timeframes favor exploitation, a pattern well-documented in reinforcement learning studies [9]. Such theoretical developments anchor experimental observations within a unified decision-making perspective, underscoring the pivotal influence of altered dopamine signaling across diverse psychiatric conditions [52, 70].

The experimental tasks employed—bandit tasks, reversal learning, and foraging exercises—each probe distinct dimensions of this equilibrium. Bandit tasks are tailored to evaluate decision-making amid uncertainty, requiring individuals to balance the benefits of exploiting familiar rewards against the potential of exploring uncharted options, rendering them highly responsive to fluctuations in reward predictability. Reversal learning tasks, by contrast, emphasize cognitive adaptability, as participants must adjust their choices in response to changing reward patterns, thereby assessing their capacity to draw on prior knowledge while remaining flexible in new situations. Foraging scenarios, with their evolving environmental demands, similarly gauge the ability to modulate exploration and exploitation in real time as circumstances shift. These paradigms place differing demands on the exploration–exploitation interplay, and their varying susceptibility to disruptions in decision-making—particularly in psychiatric contexts—highlights the importance of comparative task analysis in pinpointing specific impairments [9, 52, 70].

A methodical comparison of these tasks enables researchers to identify which elements of the exploration–exploitation continuum are most prone to disturbance across different disorders. For instance, individuals with Parkinson’s disease may exhibit shortcomings in reinforcement learning, yet they can adapt to fluctuating reward environments when dopaminergic function is appropriately adjusted [21, 70].

In all mentioned tasks, underlying mechanisms and modules of decision making are intertwined, i.e. reward sensitivity, information seeking, memory of past rewards, risk management, loss aversion and etc. are working together to get to a decision. Dissociation of these elements are not always possible. Different studies have different approaches in analysis of participants behavior. Sometimes just a behavioral index is reported such as win-stay or lose-shift rate. Sometimes multiple model-based parameters are reported such as learning rate and inverse temperature. Sometimes reported measures are contaminated and are a mixed report of multiple processes. A methodical comparison of these tasks enables researchers to identify which elements of the exploration–exploitation continuum are most prone to disturbance across different disorders. For instance, individuals with Parkinson’s disease may exhibit shortcomings in reinforcement learning, yet they can adapt to fluctuating reward environments when dopaminergic function is appropriately adjusted [21, 70].

By comparing task paradigms, we can assess whether deficits are more pronounced in situations of uncertainty (e.g., bandit tasks) than in those requiring cognitive flexibility (e.g., reversal learning), and hence yield information regarding the mechanisms and allow for more targeted treatments to be constructed. Standardisation of the measures and tasks is important in order to improve comparability across studies and translational utility of the research.

Theoretical integration: neural mechanisms

In decision making and reinforcement learning tasks, participants should decide between relying on current knowledge on some options and reducing current uncertainty about other options. Information seeking for uncertainty reduction is one of the main factors that drives exploration. So the relative amount of confidence about the options’ values plays a crucial role in exploration–exploitation balance. Dissociation between directed exploration and random exploration also emphasizes on the type of information is sought.

On the other hand, we know that diffusive neuro-modulatory systems such as Dopamine, Norepinephrine, and acetylcholine represent the uncertainty in the brain [32]. Considering that the synthesis pathway of dopamine is common with norepinephrine and the similarities of their role in the representation of uncertainty, an integrated picture of observed patterns in exploration–exploitation-trade-off in disorders can emerge. Norepinephrine that represents the unexpected uncertainty [90] is important in alertness and anxiety related disorders. So, the effects of stress and anxiety on exploration can be linked to uncertainty via norepinephrine. Dopamine also represents uncertainty and has a similar pathway for synthesis to norepinephrine, so it has similarities to attention and learning. In ADHD, ASD, Parkinson's, substance use, eating disorders, and gambling disorders, the modified dopaminergic activity plays an important role in the change of exploration–exploitation trade-off.

These models illuminate how rising uncertainty tends to steer behavior toward exploration, whereas shrinking timeframes favor exploitation, a pattern well-documented in reinforcement learning studies [9]. Such theoretical developments anchor experimental observations within a unified decision-making perspective, underscoring the pivotal influence of altered norepinephrine and dopamine signaling across diverse psychiatric conditions [52, 70].

Dopamine and balancing exploration and exploitation

Psychiatric disorders that increase exploration often involve heightened uncertainty sensitivity, impaired reward processing, or dysfunctional cognitive control, all of which influence dopamine-mediated decision-making. Anxiety disorders, for instance, enhance exploration due to an aversion to uncertainty, linked to hyperactivity in the anterior insula (aINS) and dorsal anterior cingulate cortex (dACC), which modulate exploratory decisions [1]. Similarly, depression is associated with increased exploration, as diminished reward sensitivity leads individuals to undervalue potential gains, prompting a shift toward seeking new information [16]. Additionally, schizophrenia is characterized by deficits in uncertainty-driven directed exploration, linked to dopamine dysfunction and altered novelty processing in prefrontal and striatal regions [19, 57].

Conversely, other psychiatric conditions increase exploitation by biasing individuals toward known choices. Addiction, particularly substance use disorders, is linked to a dysregulated dopaminergic system that reinforces habitual behaviors while diminishing the ability to shift strategies [4, 73]. Gambling disorder similarly reduces uncertainty-driven exploration due to impaired dopamine-related neural circuits, affecting decision-making in risky contexts [86]. In Parkinson’s disease, dopamine depletion leads to an excessive reliance on exploitation, as patients struggle with flexible reinforcement learning and exhibit perseverative choices despite changing rewards [70]. Lastly, individuals with schizophrenia who display negative symptoms exhibit decreased exploration, potentially due to motivational deficits linked to impaired ventromedial prefrontal cortex (vmPFC) and striatal function [71]. These findings underscore the critical role of dopamine and frontostriatal circuits in modulating the exploration–exploitation trade-off across psychiatric disorders.

Effect of age on exploration–exploitation trade-off

Though most studies are age-matched, but a crucial factor to consider is how age influences the exploration–exploitation balance. Adolescents, for example, tend to favor novelty-seeking and exploration-driven behavior due to ongoing brain development, especially in areas like the prefrontal cortex and striatum [38, 43, 47]. This age-related sensitivity contributes to the exploration-heavy patterns seen in disorders like anorexia nervosa, which often emerges during adolescence [48, 64].

On the other hand, older adults typically lean toward exploitative choices, reflecting changes in dopamine function and a preference for stability over new experiences. These patterns align with observations in conditions like Parkinson’s disease, where individuals often show a strong tendency toward exploitative decision-making. While research has included age-matched comparison groups, interpreting findings about age-related trends requires care. For example, people with neurodevelopmental disorders such as ADHD and ASD often display heightened exploratory behavior [6, 33]. In contrast, older individuals with these conditions may face cognitive decline that intensifies their reliance on exploitative approaches.

Clinical implications and personalized therapies

The exploration–exploitation framework carries important implications for creating tailored treatments for disorders linked to dopamine dysfunction and related illnesses. For instance, gambling disorder, marked by flawed risk-taking and reward processing, suggests that therapies aimed at improving striatal and amygdala activity could restore balance to decision-making abilities [34, 35]. Likewise, neurodevelopmental conditions such as ADHD may benefit from personalized cognitive training designed to reduce excessive and ineffective exploration caused by corticostriatal issues [69].

All interventions can be medication, behavioral and game, or stimulation and etc. can be tailored by considering their effects on dopamine, norepinephrine, confidence and uncertainty, and brain plasticity. Recent progress in neuromodulation techniques—like transcranial magnetic stimulation (TMS) and transcranial direct current stimulation (tDCS)—offers potential for targeting brain areas tied to decision-making difficulties, including the dorsolateral prefrontal cortex (dlPFC) and anterior cingulate cortex (ACC) [31, 40, 79].

The exploration–exploitation framework highlights opportunities for tailored treatments that focus on brain regions tied to decision-making. In conditions like obsessive–compulsive disorder (OCD), where faulty fronto-striatal pathways lead to rigid thinking and repetitive actions, therapies targeting these areas could prove helpful [59]. The dorsolateral prefrontal cortex (dlPFC) and anterior cingulate cortex (ACC), both essential for managing the choice between exploration and exploitation, stand out as key areas for neuromodulation efforts aimed at improving decision-making [23].

Similarly, for addictive disorders like gambling, treatments that adjust activity in the amygdala and insula could strengthen learning and reward processing [34]. Long-term studies are needed to observe how exploration–exploitation patterns evolve over time, offering better understanding of how age shapes these dynamics. Such work could uncover valuable links between age, specific disorder-related challenges, and decision-making tendencies, guiding the design of future treatments.

Addressing inconsistencies

Variations in research on psychiatric disorders and the exploration–exploitation balance stem from differences in study methods, sample diversity, biological factors, and reporting biases. Experimental setups—like bandit tasks and reinforcement learning models—vary in design and reward setups, making direct comparisons tricky. Due to implicit measures in some papers, it is possible that other non-included articles also have exploration–exploitation related measures that not captured in our early screening, especially because of our limitation on searching just one database, Pubmed. Participant differences, such as medication use, symptom intensity, and co-occurring conditions, also contribute to uneven findings. Biological elements, like dopamine function, and external factors, such as stress or sleep, further influence decision-making. To add some information for this part, we included some studies too that were on disorder related traits. Though this makes the exclusion criteria less stringent, but added more related information to our review. Moreover, publication biases may tilt the literature toward notable results. To tackle these issues, standardizing study designs, using well-defined participant groups, accounting for biological influences, and boosting transparency through pre-registration and open data sharing are practical steps forward.

Limitations and future directions

Although our review focused on age-appropriate comparisons within the selected studies, we acknowledge the need for a more nuanced examination of how age interacts with exploration/exploitation behavior across the lifespan. Future research should incorporate broader age ranges and longitudinal methodologies to explore these dynamics in greater depth and to understand their implications for clinical interventions. Additionally, computational models that integrate reinforcement learning principles with ecological validity could offer a promising direction for strengthening the robustness of findings in this area. Such models could provide more precise and dynamic representations of how exploration and exploitation behaviors evolve across different ages.

Furthermore, neuroimaging meta-analyses have the potential to clarify the neural substrates underlying exploration–exploitation dynamics, refining both diagnostic criteria and therapeutic approaches [56]. While PubMed served as the primary database for this review, we recognize that it may not fully capture the interdisciplinary nature of this topic. Due to time constraints and the scope of the review, we focused on PubMed because it provides comprehensive coverage of studies central to our research question. However, other databases might have yielded additional relevant studies, and including a broader range of interdisciplinary sources in future updates would likely enrich the findings and provide a more holistic understanding of the subject. Several additional limitations warrant explicit discussion. Publication bias is a critical concern, as studies reporting null or inconclusive findings may be underrepresented in the literature, potentially skewing our understanding of task sensitivity and disorder-specific deficits. This bias limits the generalizability of conclusions and underscores the need for greater transparency and accessibility of all research findings, regardless of their outcomes.

Moreover, heterogeneity in task designs, including differences in paradigms, outcome metrics, and experimental protocols, presents challenges for cross-study comparisons. Such variability may obscure task-specific sensitivities and hinder efforts to synthesize findings across diverse studies. Additionally, the limited number of studies addressing specific psychiatric disorders restricts our ability to comprehensively map exploration–exploitation imbalances to particular conditions. Disorders that are less frequently studied may lack adequate representation, resulting in gaps in our understanding of disorder-specific mechanisms.

To overcome these limitations, we propose the development and adoption of standardized experimental designs in future research. This includes consistently using well-defined metrics, harmonizing task paradigms, and ensuring uniform reporting standards. Standardization would enhance comparability across studies, improve reproducibility, and strengthen the translational value of research findings. Furthermore, by systematically examining the sensitivity of different tasks to exploration–exploitation deficits, standardized approaches can help clarify which paradigms are most effective in capturing specific behavioral disruptions in various disorders. Incorporating these measures will allow future research to address publication bias, reduce variability in study designs, and better represent under-studied populations. Such advancements would provide a more reliable foundation for identifying disorder-specific mechanisms underlying exploration and exploitation behaviors and support the development of targeted clinical interventions.

Conclusion

Psychiatric disorders affect the exploration–exploitation balance in subtle yet related ways, reflecting wider cognitive and brain-based challenges. By grouping these conditions into addiction-like, mood-psychiatric, and neurological categories, this review offers a cohesive model that ties together varied findings and tackles research inconsistencies. We showed the importance of uncertainty handling in the brain and disorders as a key to understand exploration–exploitation balance in disorders. Uncertainty is represented by dopamine and norepinephrine and their modulation impacts the exploratory behavior. This common theme across studies provides insights that open doors to refining personalized therapies, connecting basic research with clinical practice, and enhancing our grasp of decision-making in mental health.