Design: evaluation loop — performance data as reward signal

[gidutz]

Adtech has unusually clean ground truth: ROAS, CTR, fill rate, viewability are measurable within hours. We should exploit this.

Components:

• Performance data ingest (from ad platforms back into our store)
• Reward signal definition per agent role (what does "good" mean?)
• Offline eval harness (replay historical decisions)
• Online eval / shadow mode (run agent in parallel with a human, compare outcomes)
• Regression tracking across agent versions

Open questions:

• Attribution lag — how long do we wait before scoring a decision?
• Confounders (seasonality, exogenous events, creative fatigue)
• Per-account vs. global eval

Deliverable: eval harness spec plus initial metric definitions.