Concept: Rewards¶

What a Reward Signal Is¶

A reward signal maps the current environment state to a per-agent scalar. It is the incentive structure of the experiment — the thing that tells agents what behavior is desirable.

Every reward in this system now flows through one path — the reward_fn pipeline of RewardFunction plugins. There are two kinds of plugin: 1. The base reward — the canonical capture / step-cost / obstacle signal, supplied by the BaseReward plugin. 2. Shaping rewards — additional pluggable signals that layer on top.

The Base Reward (the `BaseReward` plugin)¶

The base reward encodes the environment's core incentives. Its values are computed by GridWorldEnv.base_reward():

Event	Agent	Value
Captures a prey (this step)	Predator	`+100.0`
Is captured (this step)	Prey	`−100.0`
On an obstacle cell	Any	`−200.0`
Each timestep	Predator only	`−5.0`

The obstacle penalty (−200) exceeds the capture bonus (+100) to strongly discourage wall-hugging. The step cost (−5) applies only to predators — it incentivizes faster capture. Prey pay no per-step cost; the SurvivalReward shaper can give prey a positive per-step signal instead.

How it is applied. GridWorldEnv.step() applies no reward on its own — rewards starts at zero for every agent, and the only source is reward_fn. run_from_config.build_environment() adds the BaseReward plugin (registry key "base") to that pipeline when rewards.base.enabled is true, and shaping plugins after it. BaseReward.compute() just calls env.base_reward() and scales by its weight.

This is the fix for issue #32. Because the base reward has a single application path (the plugin, added once), it can never be double-counted, and rewards.base.enabled genuinely turns it on or off. Do not also list base in the shaping section — that would add it twice. See DQN Variants and the Training Loop.

Shaping Rewards (Pluggable)¶

Shaping rewards are RewardFunction subclasses that provide dense guidance toward desired behaviors. They do not replace base rewards; they are added on top.

Why Shaping Matters¶

Without shaping, predators only receive signal when they capture prey (+100). In a 10×10 grid, a random agent rarely captures anything — the signal is so sparse that tabular Q-learning takes thousands of episodes to converge. Distance-based shaping provides gradient every step: moving closer → less negative reward.

Available Shapers¶

Key	Class	Signal	For
`predator_distance`	`PredatorDistanceReward`	`−weight × dist_to_nearest_prey`	Predators
`survival`	`SurvivalReward`	`+weight` per step alive	Prey

Reward Composition¶

All reward is summed in one closure, env.reward_fn, built by run_from_config.py:

total_reward[agent] = sum(plugin.compute(env)[agent] for plugin in reward_fns)

# reward_fns = [BaseReward]  (if rewards.base.enabled)  +  configured shaping plugins

gridworld.step() starts every agent at 0.0 and adds only reward_fn's output (rewards[k] += custom.get(k, 0.0)) — it contributes nothing itself. Adding a shaping component means adding an entry to rewards.yaml's shaping list; the base reward is toggled with rewards.base.enabled. No code change is needed for either.

Limitation: The current composition gives no visibility into individual components during training. If you need per-component logging, you must modify the closure or add a wrapper.

Reward Design Trade-offs¶

Trade-off	Consideration
Dense vs. sparse	Dense (shaping) accelerates learning but may cause reward hacking
Symmetric vs. asymmetric	Predator and prey rewards are on different scales (+100 capture vs +1 survival)
Base reward weight	Scaling base rewards via `weight` also scales the obstacle penalty and step cost
Shaping weight tuning	High `predator_distance` weight may cause predators to ignore obstacles

Reward as an Experiment Variable¶

The reward system is a first-class experiment variable. A common ablation study:

Condition	rewards.yaml
Sparse only	`base` reward only
+ Predator shaping	`base` + `predator_distance`
+ Survival shaping	`base` + `survival`
Full shaping	`base` + `predator_distance` + `survival`

Each condition is a YAML change. No code changes required.

MARL Reward Considerations¶

Credit Assignment¶

When multiple predators share a cell with a prey, all of them receive the full +100.0 bonus — the reward is not divided. This is a deliberate choice: in independent Q-learning each agent's Q-table is updated separately, so splitting credit would require knowing which predator "caused" the capture, which is undefined in simultaneous movement.

The consequence: cooperative multi-predator strategies are rewarded more than solo captures (the team collectively earns more), which creates natural incentive alignment without any explicit coordination mechanism.

Non-Stationarity¶

Because each agent's Q-table changes every episode, the effective reward signal seen by any single agent is non-stationary: the same observation can lead to different outcomes depending on what the other agents do. This violates the stationarity assumption that standard Q-learning convergence proofs rely on.

In practice: IQL often works but may oscillate or converge to suboptimal joint policies. See concepts/marl.md for details.

Competitive vs. Cooperative Dynamics¶

The predator-prey setup is mixed competitive-cooperative: predators cooperate implicitly (multiple predators in the same region increase capture probability) while competing with prey. Reward shaping must account for both sides:

Predator shaping (e.g., predator_distance) speeds predator learning at the cost of potential reward hacking (predators orbit prey without capturing)
Prey shaping (e.g., survival) gives prey a gradient to evade even in sparse-capture episodes

The Reward Function Interface¶

Every reward function: 1. Inherits from RewardFunction in rewards/base.py 2. Implements compute(env) → Dict[str, float] 3. Accepts weight and optional kwargs at construction 4. Is read-only — must not modify env or agent state

For the full contract, see specs/reward-function-spec.md.