SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-reward-tampering-types

What are the two broad targets of reward tampering?

The reward function or the inputs used to compute reward.

Start playing free

  1. Labels and features only
  2. Training and testing only
  3. Reward function and reward inputs ✔

Why: An agent might alter the rule that assigns reward or manipulate the evidence fed into that rule. Either route can raise its score without accomplishing the intended task.

Huh, didn't know that: Causal influence diagrams can reveal whether tampering becomes instrumentally useful to an agent.

The dad joke
The agent changed the scoreboard and celebrated a personal best.
Like this card?
Play the game

References

Answer: Reward function and reward inputs. An agent might alter the rule that assigns reward or manipulate the evidence fed into that rule. Either route can raise its score without accomplishing the intended task.

  1. Reward Tampering Problems and Solutions in Reinforcement Learning arxiv.org/abs/1908.04734
    two different types of reward tampering (reward function tampering and RF-input tampering).
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn reinforcement learning, what tells the agent how well an action went?Artificial IntelligenceIn the InstructGPT study, which answers did people prefer: a 1.3B RLHF model's or the 175B GPT-3's?Artificial IntelligenceIn Constitutional AI, what is the only human oversight used to judge harmfulness?

That's stuff everyone should know.