Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

Introduction

Online exploration is a challenging yet central problem in reinforcement learning, where the agent must traverse the environment to seek out novel states with potentially high reward values. Previous methods tend to place equal weights across all states in the prior. This is suboptimal as some states naturally grant the agent more control over what it could do (e.g., picking up a key lets the agent open a door to a new room). A natural way to characterize this idea is to consider empowerment (Klyubin et al., 2005), a metric that measures to what degree an agent can influence its future states via its actions. Because high-empowerment states have more future outcomes, it seems reasonable to explore more around these states as an agent has the potential to discover more future outcomes or behaviors.

Luckily, in many practical environments, we have prior data in the form of unlabeled trajectories. But estimating empowerment purely from offline data is difficult: existing empowerment estimators rely on online samples where a skill-conditioned agent must interact with the environment during the optimization. How can we learn empowerment from offline data to improve online exploration? Our key contribution is an offline estimator of empowerment: by learning an empowerment-maximizing policy in tandem with its discounted state occupancy, we can extract an estimator of the empowerment from prior data alone. We show that our method produces qualitatively accurate empowerment maps, and that the offline-extracted empowerment estimate can be used to improve exploration in online tasks.

Defining Empowerment

Empowerment (Klyubin et al., 2005) quantifies the maximal influence an agent can exert over its future outcomes from a given state, typically measured as the channel capacity between a control or intention and a future outcome. We use the definition from Gregor et al. (2017), which defines the empowerment in state \(s\) as the maximum possible mutual information between a uniformly-distributed random latent skill \(z \in \mathcal{Z} = \{1, \dots, K\}\) and the resulting future state \(s^+\), conditioned on the current state:

We call this the potential or maximal empowerment, to distinguish it from the effective empowerment defined next. Intuitively, empowerment should be high when a policy can realize a diverse set of outcomes where each outcome corresponds to a distinct skill. For example, an agent stuck in the dead end of a maze has very low empowerment, as its actions have almost no outcome.

While potential empowerment represents the theoretical maximum control, evaluating it directly would require searching over all policies. Instead, we can define the effective empowerment as a lower bound induced by a specific, known policy \(\pi\):

where the equality uses that \(z\) is uniform; \(\tilde{\mathcal{E}}(s;\pi) \le \mathcal{E}(s)\), with equality for the maximizing \(\pi\). Our method iteratively discovers a policy to maximize this effective empowerment, thereby approximating the potential empowerment \(\mathcal{E}(s)\), using samples \((s, a, s')\) from an offline dataset \(\mathcal{D}\).

An Algorithm for Offline Empowerment

Learning the Occupancy Measure

Because \(s^+\) is drawn from a geometric mixture over future time steps, the occupancy measures satisfy a Bellman-style recursion (Dayan, 1993):

This recursion can be used as a temporal-difference target to learn occupancy measures from off-policy transitions \((s,a,s')\), without rollouts of \(\pi\) itself. We learn \(Q^z_\theta(s^+ \mid s,a) \approx p^z(s^+ \mid s,a)\) and \(V^z_\theta(s^+ \mid s) \approx p^z(s^+ \mid s)\) from transitions \((s, a, s') \sim \mathcal{D}\), paired with a random state \(s^+\) drawn uniformly from \(\mathcal{D}\). Since a random \(s^+\) almost never equals \(s\), we add a second loss where \(s^+ = s\), so that its target includes the \((1-\gamma)\) term:

where \(\ell\) is a clipped linear exponential regression loss and \(Q_{\bar\theta}, V_{\bar\theta}\) are target networks. To tractably model these high-dimensional distributions, we parameterize the log-occupancies as scaled negative Euclidean distances between learned encoders in a shared representation space of dimension \(d\):

This formulation allows us to sample directly from the discounted state-occupancy measure as an isotropic Gaussian in the latent space \(p(\psi \mid \phi) = \mathcal{N}(\phi, \frac{d}{2}I)\), which is needed to compute the empowerment.

Policy Optimization via the Empowerment Gradient

Because empowerment depends only on the state, we need a way to describe how a skill's action distribution influences it. We define \(\mathcal{M}^{z'}(s^+ \mid s,\pi,z)\) as the marginal modified discounted state occupancy measure, representing the distribution over \(s^+\) if skill \(z\) draws its action at \(s\) from \(\pi(\cdot \mid s, z)\) and all skills follow the current policy thereafter:

We can then define the \((\pi,z)\)-modified empowerment,

which quantifies the local change in empowerment caused by changing the action distribution of skill \(z\) at \(s\). Increasing this encourages the skill-conditioned policy to take actions that make its future state occupancies under different skills distinguishable. Since we can sample from \(V^{z'}\) via the Gaussian parameterization above, we estimate \(\mathcal{E}_\Delta\) by importance-weighted Monte Carlo sampling. The complete policy objective is:

where \(\mu_\theta(s,z)\) is the policy mean and \(\alpha_{\text{bc}}\) is a hyperparameter that enforces a behavioral constraint. We train the \(Q\), \(V\) networks and the policy jointly, taking a gradient step on their combined losses at every iteration.

Estimating the Empowerment

Once training has converged, the learned policy \(\pi\) approximately maximizes the effective empowerment, so its effective empowerment \(\tilde{\mathcal{E}}(s;\pi)\) serves as an estimate of the potential empowerment \(\mathcal{E}(s)\). We compute it by substituting the learned occupancy model \(V^z_\theta\) and approximating the expectation by sampling \(M\) latents per skill, \(\psi_{z,m} \sim \mathcal{N}\big(\phi_V(s,z), \tfrac{d}{2} I\big)\):

Experiments

We answer two questions:

1. Does our method recover meaningful empowerment estimates from offline data?
2. Can we use the offline empowerment estimator to improve online exploration?

Empowerment Along Trajectories

Each video replays a trajectory from the offline dataset. The agent's path (for Cube, the gripper's) is colored by its estimated empowerment, from purple (low) to yellow (high), using the same colors as the plot below the video.

Offline Empowerment Plots

Figure 1 shows the empowerment recovered by our offline estimator on three OGBench environments; in each case it matches what we intuitively expect. On antsoccer-arena-navigate-v0 it peaks immediately around the ball in both plots, where the ant can freely choose whether to approach, dribble, or move away from it. On antmaze-medium-navigate-v0 it peaks at the maze's two corridor junctions and falls off in dead ends and corners. On pointmaze-teleport-navigate-v0 it collapses at each one-way teleporter entrance, where the agent has no control over its random next state.

(a) Ant-Soccer Arena

(b) AntMaze

(c) PointMaze-Teleport

Online Exploration

We use the empowerment estimator \(\hat{\mathcal{E}}(s)\), trained on \(\mathcal{D}\) and then frozen, to guide exploration on online, sparse-reward, goal-reaching tasks in the same environment. We add an empowerment bonus to the policy objective of a contrastive RL (CRL) agent (Eysenbach et al., 2022): a network \(B_\omega(s,a)\) predicts the maximum empowerment reached anywhere along the trajectory containing \((s,a)\), so actions that lead to high-empowerment states are rewarded, while leaving them is not penalized. Because the bonus only adds a term to the policy objective, it can also be combined with other exploration methods such as RND (Burda et al., 2019) and SUPE (Wilcoxson et al., 2025).

Adding the empowerment bonus improves over baseline CRL on all three tasks, enabling faster learning and higher performance. It also improves over RND alone when combined with it, and speeds up learning when added to SUPE on cube and antmaze.

Conclusion

In this work we introduced a method for estimating empowerment purely from offline data, by jointly learning an empowerment-maximizing policy together with its discounted state-occupancy measure. Unlike prior methods, which rely on online rollouts of a skill-conditioned policy, our estimator trains on unlabeled offline trajectories and recovers an intuitive empowerment landscape across OGBench environments. Our results show that the estimator can be used to improve online exploration on sparse, goal-conditioned RL tasks. Beyond exploration, our estimator is general, and we hope future work will apply it to other problems and domains.

BibTeX

@article{garg2026offlineempowerment,
  title   = {Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration},
  author  = {Garg, Ishir and Myers, Vivek and Li, Qiyang and Ji, Catherine and Eysenbach, Benjamin and Levine, Sergey},
  journal = {arXiv preprint arXiv:2610.24461},
  year    = {2026}
}

References

Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by Random Network Distillation. 2019.

Dayan, P. Improving Generalization for Temporal Difference Learning: The Successor Representation. 1993.

Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. Contrastive Learning as Goal-Conditioned Reinforcement Learning. 2022.

Gregor, K., Rezende, D. J., and Wierstra, D. Variational Intrinsic Control. 2017.

Klyubin, A. S., Polani, D., and Nehaniv, C. L. Empowerment: A Universal Agent-Centric Measure of Control. 2005.

Wilcoxson, M., Li, Q., Frans, K., and Levine, S. Leveraging Skills From Unlabeled Prior Data for Efficient Online Exploration. 2025.