When are LLMs sufficient policy optimizers?

Introduction

A central goal of reinforcement learning (RL) is to produce a policy that maximizes expected return with respect to a reward function [1]. To this end, an influential line of RL research has focused on developing algorithms to train a policy that maximized expected return through continued interaction with an environment [2, 3, 4, 5, 6, 7, 8]. These methods are often computationally intensive—requiring substantial interaction with the environment—and brittle, as they depend on careful hyperparameter tuning and the selection of an appropriate RL algorithm. As a result, applying RL in practice can be challenging.

To address these problems, we investigate when Large Language Models (LLMs) can produce a policy that maximizes expected return as a drop in replacement for standard RL algorithms. More broadly, we argue that LLM-based optimization can serve as a competitive alternative to standard RL algorithms for many canonical tasks, and should be considered a practical addition to the policy optimization toolbox.

To support this claim, we study a method that prompts an LLM(For this work, we use Gemini 3 Pro.) with a Python-formatted description of the state space, action space, and reward function, and asks it to implement a policy that maximizes expected return. We first sample $N$ candidate policies from the LLM. We additionally prompt the LLM to generate auxiliary evaluation criteria beyond the provided reward function. The generated policies are then rolled out and evaluated using both the reward function and these auxiliary metrics. We provide the LLM with the best-performing policy and its evaluation results, and iteratively prompt it to produce improved policies. We refer to this iterative procedure as Prompted Policy Optimization (PromptPO). Across a suite of both canonical and previously unseen environments, we then investigate when PromptPO is a desirable alternative to standard RL algorithms.

PromptPO matches or exceeds the performance of the standard RL algorithms we conisder on several delayed-reward environments that require substantial exploration, including gridworld MDPs with randomly generated transition dynamics and Point Maze, a challenging exploration environment. It achieves this performance with significantly fewer environment interactions—on average, more than over an order of magnitude fewer in the gridworld environments and $7\times$ fewer in Point Maze—suggesting it can design policies that effectively handle non-trivial exploration. Notable, for grid-world environments, PromptPO implements planning algorithms like Dijkstra's algorithm or Value Iteration using an empirical estimate of the environments transition dynamics—where the ground-truth transition dynamics model is unknown to PromptPO—highlighting its ability to select and apply a canonical planning algorithm when appropriate rather than outputting a rule based policy.

On a suite of 3 robotic manipulation tasks from Metaworld, PromptPO matches the performance of the standard RL algorithms we conisder using, on average, more than $7\times$ fewer environment steps, and in a 4th Metaworld environment, PromptPO outperforms the final performance of the standard RL algorithms within the training budgets we consider. We additionally evaluate PromptPO in three real-world control tasks: designing lockdown regulations for the COVID-19 pandemic, administering insulin to patients with type II diabetes, and controlling a fleet of autonomous vehicles merging onto a highway. PromptPO is more than $60\times$ more sample efficient in the former and outperforms the final performance of standard RL algorithms we conisder within the training budgets we consider in the latter two environments. PromptPO underperforms standard RL algorithms on a suite of MuJoCo continuous control tasks, indicating that current LLMs may have difficulty designing policies for environments that require fine-grained control.

We demonstrate that PromptPO, by way of reasoning over natural language descriptions of the state space, action space, and reward function, can produce policies that maximize the expected return with significantly fewer environment interactions—and algorithmic hyperparameters to tune—than standard RL algorithms in a diverse suite of environments. To maximize expected return, and without further explicit prompting, the policies PromptPO outputs range from tuned proportional controllers or rule-based plans to policies that run planning algorithms like value iteration. Broadly, we show that LLMs are sufficient black box policy optimizers for many environments and should potentially been seen as practical substitutes for canonical RL algorithms.

We conclude by positioning PromptPO within the RL landscape and discussing its implications for practice and research:

• RL practitioners should consider PromptPO as an easy-to-use first attempt to use RL.
• RL researchers should evaluate their methods in settings where strong priors are available, whether over the environment, policy class, or optimization process; LLM-based policy optimization methods should be supported in standard RL libraries.
• Will LLM-based optimization go beyond the capabilities of current RL algorithms by implementing novel RL algorithms?

We release our code base here, including the minimal implementation of PromptPO used for this work, the context we provided to PromptPO for each environment, and examples of the policies PromptPO generates for each environment.

Related Works

We follow the same procedure as a line of prior work that prompts an LLM to iteratively generate a code artifact [9, 10, 11] or solution [12, 13, 14] to optimize for a specific objective. We investigate when this framework is sufficient to solve classical, single agent sequential decision making tasks as a substitute for standard RL algorithms, and in particular how this framework compares to standard RL algorithms with respect to environment sample efficiency. Following the insights from [15] that policies can be represented as executable programs, we represent generated policies as Python code. Our work differs from [15] and a line of others [16, 17, 18, 19, 20] in that they use LLMs to generate policies to satisfy a natural language description of a task or reward function. Rather, we study the setting where an LLM must produce a policy to optimize expected return with respect to a specified reward function, i.e., the standard RL setting. We summarize the differences between our work and these in Table 1.

Other work focuses more directly on using LLMs for policy optimization, such as prompting an LLM to output an action directly at every decision making step [21, 22]. This approach doesn't naturally scale to the long-horizon decision making settings we consider. [23] uses an LLM to generate policy parameter vectors, but they assume the policy can be expressed with a small, human-specified set of parameters that the LLM can reliably output, limiting expressivity and requiring manual design of the policy class. We investigate the more general setting where an LLM must output the policy itself, not parameters for a prespecified policy class.

The primary objective of this work is to investigate when PromptPO serves as a strong alternative to standard RL methods, and to argue that LLM-based policy optimization should be adopted as a standard tool within the RL toolkit.

Prompted Policy Optimization (PromptPO)

In this section, we describe a specific instantiation of Prompted Policy Optimization. More broadly, we use this term to refer to a class of methods that leverage LLM-based optimization to generate policies for sequential decision-making tasks. The particular method presented here serves as a concrete implementation to evaluate the effectiveness of this broader class.

PromptPO inputs a description of the state space, action space, and the reward function implementation. These inputs are expressed in Python code, and naturally accompany environments designed for RL such as via Gymnasium [24] interfaces. Importantly, we avoid providing context about the environments' transition dynamics to understand PromptPO's ability to maximize expected return without explicit model based knowledge. All context used by PromptPO is available in our Github repository.

We first ask PromptPO to implement an evaluation class called Feedback via the prompt summarized in Figure 9. The implemented evaluation provides additional supervision beyond the environment reward function. Standard RL algorithms reason about prior interactions by way of Bellman backups or temporal difference errors propagated through the state-action space. In contrast, PromptPO updates policies using only trajectory-level feedback, making the design of evaluation functions beyond the reward signal a key component of the method. While future work could incorporate state–action level feedback into PromptPO, we hypothesize that current LLMs may struggle to reason over long-horizon sequences of individual transitions, particularly in environments with hundreds or thousands of steps. Upon generating an evaluation function, PromptPO is then prompted to generate a policy to maximize expected return using a prompt summarized in Figure 10.

We sample $N$ candidate policies from PromptPO and roll out each policy in the environment. For each rollout, we compute the return under the provided reward function and evaluate performance using the generated evaluation function. The best-performing policy—measured by return—is then reported back to PromptPO, which produces a natural language reflection on its performance using the prompt in Figure 11. The history of the best policy from all previous rounds is maintained in context. Finally, this reflection, along with the history of prior best policies and their associated reflections, is used to generate a new policy via the prompt in Figure 10. At any point in the iterative process, if a Python execution error occurs due to incorrectly generated code (e.g., in the policy or evaluation), PromptPO is provided with the error and prompted to re-implement the relevant code artifact.

Experiments Summary

Figure 2 and 3 illustrate a summary of our results when $N=10$ policy candidates are sampled from PromptPO at each round. PromptPO matches or outperforms the best performing RL algorithms in 15/19 environments, is substantially more sample-efficient in 14/19 environments, and is more than an order of magnitude more sample efficient in 11/19 environments.

Exploration Environments We introduce NoiseWorld, a suite of randomly generated grid worlds designed to evaluate PromptPO’s exploration capabilities while mitigating the LLM’s prior knowledge of the environment. We additionally evaluate PromptPO on a physics-based maze navigation task—the Point Maze Gymnasium environment [25]—using a custom maze layout. Across these environments, PromptPO matches the performance of the best-performing RL algorithms, demonstrating strong exploration driven by sampling and evaluating many candidate policies at each update round. Notably, 2/5 of the NoiseWorld environments remain unsolved by both PromptPO and RL baselines. Further details are provided in Section 6.1. Interestingly, in the NoiseWorld environments, PromptPO’s generated policies implement a planning algorithm, such as Dijkstra's algorithm or Value Iteration using the board layout, which is provided in the observation, together with an empirically estimated transition model derived from prior experience. The resulting planning algorithm is executed at every environment timestep. This behavior indicates the effectiveness of PromptPO's prior on what optimization procedure to employ. In Point Maze, PromptPO's implemented policy is a proportional controller.

Robotics Environments We evaluate PromptPO on six MuJoCo [26] and four Meta-World robotic manipulation environments [27]. A key distinction is that MuJoCo tasks require actions in the form of joint torques, whereas Meta-World tasks operate over end-effector displacements. PromptPO underperforms standard RL algorithms on the MuJoCo suite, but in Meta-World it achieves greater sample efficiency in all tasks and improves final performance on one task. These results highlight a limitation of PromptPO: it struggles with fine-grained continuous control (e.g., joint torques), but performs well when the action space is more amenable to natural language reasoning. See Section 6.2 for additional details. Across all robotics environments, PromptPO generates proportional controllers and tunes their parameters as it receives additional environment feedback.

Real World Control Environments We evaluate PromptPO on three real-world decision-making environments used by [28]: administering insulin to patients with Type II diabetes, determining lockdown regulations to manage a Covid-19 pandemic, and controlling the accelerations of a fleet of autonomous vehicles merging onto a highway. PromptPO is more than $60\times$ more sample efficient in the Pandemic Mitigation environment, and outperforms the final best performance of RL for the training budget we consider in the Glucose Monitoring and Traffic Control environments. These results suggest that PromptPO can be a strong policy optimizer in real-world settings, where the LLM may leverage prior knowledge about the environment from pretraining to generate effective policies. See Section 6.3 for additional details. In these environments, PromptPO implements rule-based heuristic policies that are iteratively refined as additional environment feedback is obtained.

On Pretraining Biases and Privileged Information in PromptPO Most of the environments we consider are publicly available and likely present in the LLM’s pretraining data. As a result, PromptPO may benefit from strong inductive biases or access to information about environment dynamics or effective policies that would not typically be available to an RL agent. We take several steps to mitigate these concerns. First, we evaluate PromptPO on newly generated environments (e.g., NoiseWorld) and novel configurations of existing environments (e.g., Point Maze) that are unlikely to appear in pretraining data. We do not allow access to external tools such as internet search, and we do not provide transition dynamics in context. We additionally audit generated policies using LLM-guided search to check for reliance on publicly available solutions or privileged information. In one surfaced instance, for example, PromptPO reproduced a maze layout from publicly available documentation; we mitigate this by using a custom layout. While prior exposure to related environments likely provide useful inductive biases in ways that are hard to predict, PromptPO must still synthesize policies that maximize expected return in the specific instances we consider.

On Comparing PromptPO to RL from Scratch We compare PromptPO to RL methods that optimize for expected return starting from a randomly initialized policy. This comparison is intentionally asymmetric: while RL methods learn solely from environment interaction, PromptPO leverages substantial prior knowledge from pretraining. Our goal is not to equalize these settings, but to understand the extent to which pretraining can substitute for environment interaction in canonical RL tasks. In particular, this comparison targets a central question: how much useful structure about policies and environments is already captured by large-scale pretraining, and how effectively can it be leveraged for decision making? From a practical perspective, we argue that such comparisons are appropriate, as practitioners can choose between methods that learn from scratch and, via LLM based optimization methods, those that use pretrained models. Our results suggest that LLM-based optimization methods like PromptPO can offer a compelling alternative due to their strong priors and resulting sample efficiency.

Detailed Environment Descriptions and Experiments

We provide additional details about the set of environments we use for evaluation in Section 4, and present results for each individual environment. In Appendix 9, we present results for PromptPO trained with different values of $N$, the number of candidate policies generated per round.

Exploration Environments

NoiseWorld We propose NoiseWorld to evaluate PromptPO in a setting where it doesn't reasonably have access to a strong prior on the environment from its training data. NosiseWorld is a grid-world environment where the agent starts in the top left corner and must maximize its expected return by navigating to a goal state in the bottom right corner. The action space consists of moving in the 4 cardinal directions. By default, each NoiseWorld board is $10\times10$ with a horizon of $100$. The agent accrues a $-1$ reward at each timestep, and $+1000$ reward for reaching the goal state. NosieWorld has additional 6 cell types:

• Cell type 0 is a blank; transitions out of cell type 0 are deterministic
• Cell type 1 is a wall; no transitions into these cells are successful.
• Cell type 2 is such that any transitions out of this cell are successful with probability $p$, and otherwise result in the agent staying in the current cell with probability $(1-p)$.
• Cell type 3 is such that any transitions out of this cell are successful with probability $p$, and otherwise result in the agent transitioning back to the start state with probability $(1-p)$.
• Cell type 4 ends the episode with a penalty of $-1000$.

We instantiate NoiseWorld1, NoiseWorld2, and NoiseWorld3 where cell types are sampled from the types listed above, and where relevant, the transition success probabilities are also sampled for each cell. When providing the observation space to PromptPO, which consists of the agents current location and the board layout, we do not indicate which cells types correspond to what transition dynamics or behaviors; PromptPO must figure this out on its own.

We additionally evaluate PromptPO with NoiseWorld4 and NoiseWorld5, which in addition to the cell types above, also have cell types 5 and 6; there is exactly one of these cell types each per board, and to attain the reward of $+1000$ upon visiting the goal state, the agent must visit the cell of type 5 and then the cell of type 6 first. Otherwise, reaching the goal state ends the episode with no bonus.

Point Maze We evaluate PromptPO in the Point Maze environment [30], where a 2-DoF ball that is force-actuated must navigate through a maze to reach a target goal. The action space is continuous and 2-dimensional, and the observation consists of the balls position and velocity, the goal's position, and whether the goal has been reached. Each episode has a horizon of 800, and a delayed reward of +1 for reaching the goal and 0 otherwise. We generate a new $10 \times 9$ Point Maze map that differs from the default map provided by [30] to force PromptPO to explore the environment rather than rely on the LLM's training data. We found that when we did not do this, PromptPO—without access to the internet—recalled the default map used by Point Maze Large from their documentation.

Considered RL Baselines For NoiseWorld, we evaluate PPO with and without hindsight experience replay [31]—which often aids exploration—and report the mean return of the best performing method for each board layout. For Point Maze, we consider SAC, PPO, and variants of both with hindsight experience replay, reporting the mean return of the best performing method. See Appendix 8 for details on our choice of hyperparameters and implementation details for NoiseWorld. For Point Maze, we use the hyperparameters from [32].

Results PromptPO matches the best performing RL algorithms performance with significantly fewer environment samples in all NoiseWorld environments and in Point Maze; in the NoiseWorld environment, PromptPO attains its best performing policy without updating its generated policies using feedback from the environment, despite no information about the different cell types. We attribute this strong performance to PromptPO's ability to sample many different policies, which in effect results in strong exploration capabilities. When the many different policies sampled by PromptPO don't immediately yield the best performing policy—as is the case in Point Maze–PromptPO incorporates the environment feedback (e.g., expressed via the return and generated evaluation function) to sample improved policies in the next round. Figure 5 shows this learning capability for Point Maze, and Figure 4 show it for NoiseWorld.

Unlike in NoiseWorld1, NoiseWorld2, and NoiseWorld3, for NoiseWorld4 and NoiseWorld5, PromptPO and the best performing RL algorithm fail to find a policy that behaves near-optimally; learning to visit cell types 5 and 6 to unlock the high reward upon entering the goal state. This result indcates that PromptPO, like standard RL methods, struggle with difficult exploration problems; future work should try to incorporate lessons from exploration for RL into PromptPO. Providing PromptPO with context about cell types 5 and 6, i.e., that visiting them is necessary to attain the high reward bonus at the goal state, results in PromptPO producing a policy that exhibits this desired behavior. This type of prior knowledge might be difficult to specify to a standard RL algorithm, but is quite simple to specify to PromptPO via natural language and, as shown in Figure 15, enables PromptPO to generate a performant policy.

NoiseWorld1

NoiseWorld2

NoiseWorld3

NoiseWorld4

NoiseWorld5

Robotics Environments

Mujoco Environments We evaluate PromptPO in the Hopper, Ant, Half-Cheetah, and Swimmer environments, where the objective is to train a locomotion policy, and in the Reacher and Inverted Pendulum environments where the objective is to move a two-jointed robot arm close to a target and balance a two-jointed pole on a cart, respectively [26].

Metaworld Environments We evaluate PromptPO in the Button Press, Door Open, Drawer Open, and Pick and Place environments [27], where the objective is to train a 7-DOF Sawyer robotic arm to press a button, open a door, open a drawer, and pick and place objects, respectively.

For the Mujoco and Metaworld environments, we use the default observation and action spaces, and for PromptPO, additionally provide a description of what each observation variable denotes.

Considered RL Baselines For each Mujoco environments, we train a policy with PPO, SAC, and TD3 and report the mean return of the best performing method per environment. We use the hyperparameters from [33] for the Mujoco environments, and from [27] for Metaworld.

For Metaworld, we train policy with SAC, following the best performing method of [27].

Results PromptPO under performs RL in all Mujoco environments except Reacher, where it is substantially more sample efficient, and Swimmer, where it attains similar performance but after many more environment interactions. On the other hand, PromptPO is significantly more sample efficient than RL in all Metaworld environments, and additionally outperforms the final performance attained by RL in the Pick and Place environment. In Metaworld, PromptPO generates its best performing policy after incorperating only a single or a few rounds of feedback from the environment, as illustrated in Figure 6.

The difference in relative performance between PromptPO and RL in the Mujoco and Metaworld environments provide insight into the type of environments where PromptPO is a sufficient policy optimizer; for Mujoco, the action space consists of torques applied to hinge joints, while for Metaworld it is the end-effector displacement and gripper finger positions. We hypothesize that PromptPO struggles to implement a performant policy for environments that require fine-grained continuous control, and hence it performs poorly in Mujoco environments while exhibiting strong performance in Metaworld environments. For all environments, PromptPO implements a proportional controller and tunes the controller after subsequent environment feedback is provided in context. This suggests that, for some tasks, PromptPO restricts its policy class.

Button Press

Door Open

Drawer Open

Pick Place

Ant

HalfCheetah

Hopper

Swimmer

Inverted Double Pendulum

Reacher

Real World Control Tasks

Glucose Monitoring In Glucose Monitoring, the objective is to design a policy to administer insulin to a patient with Type II diabetes; an action must be outputted every 5 minutes for 20 days, and observations are provided by a continuous glucose monitor (CGM). We use the environment parameters and reward function from [28].

Pandemic Mitigation In Pandemic Mitigation, the objective is to design COVID-19 pandemic lockdown regulations to balance economic and health outcomes. The available actions are four lockdown regulation stages, and the observations consist of the states from an SEIRS pandemic model. We use the Pandemic simulator from [34] and the environment parameters and reward function from [28]

Traffic Control In Traffic Control, a policy must control a fleet of autonomous vehicles merging onto a highway to maximize traffic flow. The action space consists of the autonomous vehicle accelerations, the observation space is location and velocity—as well as other statistics derived from these measures—of all cars on the highway. We use the simulator from [35] and the environment parameters and reward function from [28]

Considered RL Baselines We train a policy with PPO in all environments, following the implementation and hyperparameters of [36] for these environments.

Results For these environments, representative of many real world control tasks, PromptPO is both significantly more sample efficient than RL and, for Traffic and Glucose environments, substantially outperforms the best policy found by RL within the environment sample budget considered. Training curves for all methods are shown in Figure 8. These results showcase PromptPO's strength as a policy optimizer in real world settings where it may be capable of leveraging its pretraining data as a strong prior on generating a performant policy.

bibliographystyle{plain} bibliography{references}

Glucose Monitoring

Pandemic Mitigation

Traffic Control

Implications of LLM-Based Policy Optimization

RL practitioners should consider PromptPO as an accessible first approach PromptPO is both easy to use—in that it requires few algorithmic hyperparameters to tune and interfaces with natural language information a practitioner can provide—and is a strong choice of policy optimizer for environments where an LLM can reason over the state and action space. Practitioners without extensive RL expertise may find PromptPO easier to apply to sequential decision-making problems than standard RL methods. While PromptPO has clear limitations—most notably its reliance on state and action representations that are amenable to natural language reasoning, which excludes settings with image or embedding-based observations—it provides a strong and practical starting point for policy optimization in many real-world environments that meet these requirements.

We should prioritize RL research beyond the "training from scratch'' paradigm A dominant paradigm in RL research is to compare methods that train policies from scratch. This approach is well-suited for developing general-purpose algorithms that can be applied to novel environments where no prior knowledge is available. However, the emergence of LLM-based optimization highlights the value of leveraging large-scale pretraining, which can provide both effective optimization strategies and strong priors over many environments. This suggests an additional research direction for the RL community: developing and evaluating methods in settings where such priors are available, whether over the environment, the policy class, or the optimization process.

For many research questions and methods, particularly those used in conjunction with existing policy optimizers—such as methods to encourage exploration, abide by safety constraints, leverage world models for planning, construct hierarchical policies, or enable continual learning—showing that the proposed techniques improve performance with both standard RL methods and LLM-based policy optimization methods would be a practical contribution. Evaluating methodological improvements may be difficult as the internet is populated with more policy optimization solutions, but environments with randomly generated transition dynamics could provide a useful testbed for LLM-based policy optimization methods.

Standard RL libraries should support LLM-Based policy optimization methods To encourage adoption of PromptPO and related LLM-based optimization methods, widely used RL libraries should support them as drop-in alternatives to standard approaches such as PPO or SAC. However, widespread adoption may also introduce new evaluation challenges: as solutions to many policy optimization problems become publicly available, it may become increasingly difficult to assess whether future methods genuinely generalize to unseen environments. To mitigate this, we advocate for the development of more unseen evaluation settings, such as environments with randomly generated transition dynamics.

Will LLM-Based policy optimization go beyond the capabilities of current RL algorithms by implementing novel RL algorithms? Proponents of the Bitter Lesson [29] might assert that the human inductive biases embedded in LLMs would limit PromptPO methods from significantly surpassing human capabilities. If so, PromptPO remains a practical tool for generating high-performing policies across many environments, and may facilitate broader adoption of RL in real-world settings—particularly among practitioners without RL expertise. But we observe that in the NoiseWorld environments, PromptPO implements Value Iteration without being explicitly prompted to do so. In the continuous control environments, like in Mujoco or Metaworld environments, PromptPO implements and tunes a proportional controller. This provides evidence that PromptPO can select appropriate policy optimization procedures for a given setting, suggesting that future LLM-based optimization methods may similarly implement more sophisticated existing RL algorithms—or even develop novel ones that extend beyond current approaches.

Acknowledgements

Simon Guo, Miguel Liu-Schiaffini, and Shannon Sequeira were extremely helpful in discussing ideas and providing feedback on this post.