Proposal: redacted run provenance manifest for reproducible evolution experiments

[xs271828]

Motivation

OpenEvolve already records detailed candidate-level information through EvolutionTrace, program metadata, checkpoints, and lineage extraction. What appears to be missing is a small run-level provenance record that ties those artifacts to the exact inputs and execution environment that produced them.

A machine-readable manifest would help users compare runs, diagnose irreproducible results, and prepare research artifacts without duplicating trace or checkpoint data.

Proposed scope

Write one redacted JSON manifest under the run output directory, for example run_manifest.json, with a versioned schema containing:

• schema_version and a generated run_id;
• OpenEvolve package version and optional source commit;
• UTC start/end timestamps and a normalized stop reason;
• SHA-256 hashes for the resolved config, initial program, and evaluator;
• declared random seed, Python version, platform, and architecture;
• model/provider names and a sanitized API origin when available;
• relative references to the evolution trace and latest checkpoint;
• a measurement-source field distinguishing observed values from configured or inferred values.

The manifest should be written atomically and updated at lifecycle boundaries so interrupted runs still retain useful provenance.

Redaction and safety requirements

The manifest should never contain:

• credential values or environment-variable contents;
• URL userinfo, query strings, or fragments;
• absolute user paths;
• raw prompts, responses, candidate code, or evaluator artifacts already governed by trace settings.

Credential environment-variable names could be recorded only if maintainers consider that useful and safe.

Non-goals

• This would not claim exact reproducibility from remote LLM services.
• It would not measure provider billing or enforce an exact USD limit.
• It would not add holdout data to the evolution loop or expose holdout results to prompts.
• It would not replace EvolutionTrace, checkpoint metadata, or EvaluationResult.
• It would not include framework-specific research workflow fields.

Compatibility

The feature could be additive and backward-compatible. Readers should ignore unknown fields, and older checkpoints should continue to load without a manifest. The CLI/library could enable it by default or behind a small configuration flag depending on maintainer preference.

Open questions

1. Should the manifest be owned by the CLI runner, the main evolution lifecycle, or a reusable utility called by both?
2. Is JSON preferable to extending existing checkpoint metadata?
3. Which stop-reason vocabulary would be stable enough for downstream tools?
4. Should model API origins be omitted entirely rather than sanitized?

If this belongs in core, I can prepare a small PR with the schema, redaction tests, lifecycle tests, and backward-compatibility coverage after the maintainers confirm the preferred ownership and file location.

[codelion]

Thanks for writing this up, and for checking the existing artifacts first. This fills a real gap. Today metadata.json is pure evolution state, the trace metadata is just created_at plus the include flags, and the resolved config is never written to the output directory at all. Once CLI overrides are applied there is no record on disk of what actually ran.

On your open questions:

1. Own it in the evolution lifecycle, not the CLI. openevolve/api.py exposes run_evolution for library use, and those runs should get a manifest too.
2. Separate JSON file. Checkpoint metadata gets read back on resume, so I would rather not mix provenance into it.
3. Keep it small and closed: completed, max_iterations, interrupted, error.
4. Omit the API origin entirely. Model and provider names are enough.

One change worth making: write the redacted resolved config rather than only a SHA-256 of it. A hash tells you two runs differ but not how, and the config is small. Hashes still make sense for the initial program and evaluator since those are user files. If you write the config, redact llm.api_key and llm.models[*].api_key, the nested one is easy to miss.

I would also drop the measurement-source field and make it default on with no config flag.

[xs271828]

Thanks — this is very helpful, and I agree with the direction.   I’ll keep this in the evolution lifecycle (run_evolution) so library users receive the same artifact, write it as a separate JSON manifest rather than checkpoint metadata, and make it default-on.   I’ll use the closed stop-reason set you suggested, omit API origin entirely, and write a redacted resolved config instead of only its hash. The initial program and evaluator will remain SHA-256 references. I’ll also explicitly cover both llm.api_key and llm.models[*].api_key in the redaction tests.   I’ll drop measurement_source and come back with a small backward-compatible PR plus tests.

…

---Original--- From: "Asankhaya ***@***.***> Date: Wed, Jul 29, 2026 10:36 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [algorithmicsuperintelligence/openevolve] Proposal: redacted run provenance manifest for reproducible evolution experiments (Discussion #475) Thanks for writing this up, and for checking the existing artifacts first. This fills a real gap. Today metadata.json is pure evolution state, the trace metadata is just created_at plus the include flags, and the resolved config is never written to the output directory at all. Once CLI overrides are applied there is no record on disk of what actually ran. On your open questions: Own it in the evolution lifecycle, not the CLI. openevolve/api.py exposes run_evolution for library use, and those runs should get a manifest too. Separate JSON file. Checkpoint metadata gets read back on resume, so I would rather not mix provenance into it. Keep it small and closed: completed, max_iterations, interrupted, error. Omit the API origin entirely. Model and provider names are enough. One change worth making: write the redacted resolved config rather than only a SHA-256 of it. A hash tells you two runs differ but not how, and the config is small. Hashes still make sense for the initial program and evaluator since those are user files. If you write the config, redact llm.api_key and llm.models[*].api_key, the nested one is easy to miss. I would also drop the measurement-source field and make it default on with no config flag. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[xs271828]

Thanks again for the guidance. I have opened a small draft implementation at #476. It owns the manifest in the evolution lifecycle, writes a separate atomic JSON file by default, records the redacted resolved config, uses the closed stop-reason vocabulary, omits API origins, and covers top-level plus per-model API-key redaction in tests. I kept checkpoint metadata and research-workflow features out of scope.