Skip to content

perf(database): create the checkpoint programs directory once per save - #478

Open
dexhunter wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
dexhunter:perf/checkpoint-save-write-loop
Open

perf(database): create the checkpoint programs directory once per save#478
dexhunter wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
dexhunter:perf/checkpoint-save-write-loop

Conversation

@dexhunter

Copy link
Copy Markdown
Contributor

ProgramDatabase.save() writes one JSON file per program, and for every one of
those files _save_program calls os.makedirs(programs_dir, exist_ok=True). At the
shipped default population_size=1000 that creates the same directory 1,000 times
per checkpoint, and 999 of those calls do nothing.

This PR creates the directory once in save(), lifts two loop-invariant lookups out
of the per-program loop, and writes each record with one f.write(json.dumps(...))
instead of json.dump(..., f), which pushes the encoded string into the buffered
text layer in fragments.

Measurement

CPU time for a single ProgramDatabase.save() call, median of 5 repetitions, each on
a freshly generated 1,000-program population, taken under an exclusive lock on an
otherwise idle machine:

ms CPU
main at 411fb59 241.1829
this branch 182.8334 and 183.7943

That is 24.0% off the call. Repeated measurements of the unmodified baseline span
238.05 to 244.01 ms across eight runs, about 2.5%, so a 58 ms difference sits well
outside the noise.

The fixture uses the shipped defaults: population_size=1000, num_islands=5 and
log_prompts=True. The last one sets the scale of the number. With prompt logging
on, ProcessParallelController records each accepted child's rendered prompt and LLM
response, and _save_program merges that record into the program's JSON, so a
realistic checkpoint carries roughly 42,000 characters of prompt per program on top
of the code. With prompts_by_program empty the same call costs 121.9 ms at the same
population size, and the absolute saving scales down with it.

What this is not

This is not a user-visible latency win, and I want to be plain about that. A
checkpoint happens once every checkpoint_interval iterations, 100 by default, so a
10,000-iteration run spends about 24 s inside save() in total and this recovers
about 5.8 s of it. What makes it worth doing is where the call sits: _save_checkpoint
runs synchronously inside the async run_evolution loop, so the time is a stall
during which no iteration is submitted and no worker result is collected.

Behaviour

The bytes on disk do not change. json.dump(d, f) and f.write(json.dumps(d))
produce identical output, and program.to_dict() is still what gets serialized. I
checked that rather than assuming it: I wrote a checkpoint from the same seeded
200-program population with and without the patch and compared every file by SHA-256,
once with prompts_by_program populated and once with it empty. Both file sets match
and all 402 digests match.

_save_program gains a programs_dir keyword argument defaulting to None. When it
is None the method derives and creates the directory exactly as before, which is
what the single-program call site in add() depends on.

python -m unittest discover tests passes, 430 tests.

Headroom I deliberately left alone

Program.to_dict() is dataclasses.asdict(self), a recursive deep copy of a dict
that gets serialized and thrown away immediately. Skipping it takes the same call to
roughly 164 ms. That cost belongs to Program rather than to the save path, and #477
already proposes a change there, so this branch does not touch it.


The direction came from an automated optimization search over openevolve/database.py
scored on the CPU cost of one save() call, with edits confined to save and
_save_program and every candidate checked byte-for-byte against an unmodified
reference. The search record is at
https://dashboard.weco.ai/share/cn8uWyk3qw1XMT0mZdeMvu1fGmKCizjI (15 scored
candidates over 21 attempts, from a 238.05 ms measured baseline down to 164.46 ms). I
rebuilt this diff by hand against current main and re-measured it, and I offer the
link as a record of the search rather than as authorship of the patch.

The search's best candidate is not what this PR proposes. It reaches 164.46 ms partly
by dropping program.to_dict() in favour of the live __dict__, replacing
os.path.join with string concatenation, and disabling check_circular on the JSON
encoder. The first collides with #477, the second breaks Windows paths, and the third
removes a safety check on user-supplied metrics and metadata. This branch keeps only
the changes that alter nothing but the cost.

_save_program called os.makedirs(programs_dir, exist_ok=True) for every
program written, so a checkpoint at the default population_size=1000 created
the same directory 1,000 times. Create it once in save() and pass it down.

Also lift the log_prompts and prompts_by_program lookups out of the
per-program loop, and write each record with a single f.write(json.dumps(...))
rather than json.dump(..., f).

The bytes written are unchanged: checkpoints from an identical seeded
200-program population compare equal by SHA-256 before and after, both with
recorded prompts and without.

One save() call at the shipped defaults goes from 241.1829 ms to 182.8334 ms
of CPU, a 24.0% reduction, measured as the median of 5 repetitions on freshly
generated populations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant