William Saunders
(This post is aiming to address the Informed oversight problem, as written about by Paul Christiano here)
Paul Christiano raises the possibility of addressing the informed oversight problem by training an agent to, given a state , choose an action and explanation to maximize the sum of the overseer’s evaluation plus a measure of how informative the explanation is with regards to the action . The hope is that this incentivizes the agent to always take good actions, and provide a good explanation of any information that is relevant to the overseers evaluation.
The problem is that in cases where providing a good explanation reduces the overseer’s evaluation of the action, the agent will tend to either take actions which can be explained well, but are not necessarily good, or to take actions that seem good and withhold explanations.
Proposal: allow the overseer to accumulate information over time, such that their evaluation functions are not stationary. Replace the function with to reflect the overseers evaluation of the action before viewing the explanation , but in light of all previous explanations. Change the function to reward only new information provided to the overseer (that it couldn’t figure out given it’s knowledge and all previous explanations).
In this setup, if there is some action that appears to be good without the explanation, but is actually bad, the agent’s best move is to present it and it’s explanation, receiving high reward for both. But if the agent subsequently tries to present the action and it’s explanation a second time, the overseer will evaluate the action and explanation poorly. Crucially, each action is a separate episode in the setup, so the agent has no incentive to perform poorly in one episode in order to perform better in some subsequent episode.
It seems like this setup removes the opposition of the incentives to explain and to produce good seeming actions. In each episode, there is no cost to improving the agent’s ability to explain itself. For any deceptive action that is explained well enough, it will stop being rewarded and the agent will stop proposing it.
For this to work, the agent’s explanations need to be “generalizable”, informative not only about the particular action, but about a larger class of actions, allowing the overseer to change their oversight procedure or modify the agent to prevent it from taking similar actions in the future. On it’s own, the overseer will have some ability to generalize from explanations for particular actions: by paying more attention to the class of deceptive strategies in the future, by examining a trace of the agent’s computation to look for signs that correlate with the deceptive strategy, etc. Other examples, for the plagarism case could include:
Caveats:
- It seems like this makes training the agent to produce explanations much more difficult, as it only gets high reward for producing each type of explanation once, and we might need a higher quality of explanation to generalize well.
- For any given deceptive strategy, there might be a period of time between when the agent can perform the strategy, and when it can explain it well enough to the overseer to allow them to properly evaluate it (even if there is no information for the agent to hide information, it might just be a harder task to explain things properly). During this time, the agent is not aligned, and it is not possible to detect that the agent is unaligned. However, if all deceptive strategies can eventually be explained, then it seems that in some sense the agent will be aligned in the limit of this process as oversight continues.
- This does not help if there are deceptive actions that cannot be explained in a generalizeable manner.
I’d appreciate feedback on this idea (especially additional problems with it)
Written with StackEdit.