perf(database): create the checkpoint programs directory once per save - #478
Open
dexhunter wants to merge 1 commit into
Open
Conversation
_save_program called os.makedirs(programs_dir, exist_ok=True) for every program written, so a checkpoint at the default population_size=1000 created the same directory 1,000 times. Create it once in save() and pass it down. Also lift the log_prompts and prompts_by_program lookups out of the per-program loop, and write each record with a single f.write(json.dumps(...)) rather than json.dump(..., f). The bytes written are unchanged: checkpoints from an identical seeded 200-program population compare equal by SHA-256 before and after, both with recorded prompts and without. One save() call at the shipped defaults goes from 241.1829 ms to 182.8334 ms of CPU, a 24.0% reduction, measured as the median of 5 repetitions on freshly generated populations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ProgramDatabase.save()writes one JSON file per program, and for every one ofthose files
_save_programcallsos.makedirs(programs_dir, exist_ok=True). At theshipped default
population_size=1000that creates the same directory 1,000 timesper checkpoint, and 999 of those calls do nothing.
This PR creates the directory once in
save(), lifts two loop-invariant lookups outof the per-program loop, and writes each record with one
f.write(json.dumps(...))instead of
json.dump(..., f), which pushes the encoded string into the bufferedtext layer in fragments.
Measurement
CPU time for a single
ProgramDatabase.save()call, median of 5 repetitions, each ona freshly generated 1,000-program population, taken under an exclusive lock on an
otherwise idle machine:
mainat 411fb59That is 24.0% off the call. Repeated measurements of the unmodified baseline span
238.05 to 244.01 ms across eight runs, about 2.5%, so a 58 ms difference sits well
outside the noise.
The fixture uses the shipped defaults:
population_size=1000,num_islands=5andlog_prompts=True. The last one sets the scale of the number. With prompt loggingon,
ProcessParallelControllerrecords each accepted child's rendered prompt and LLMresponse, and
_save_programmerges that record into the program's JSON, so arealistic checkpoint carries roughly 42,000 characters of prompt per program on top
of the code. With
prompts_by_programempty the same call costs 121.9 ms at the samepopulation size, and the absolute saving scales down with it.
What this is not
This is not a user-visible latency win, and I want to be plain about that. A
checkpoint happens once every
checkpoint_intervaliterations, 100 by default, so a10,000-iteration run spends about 24 s inside
save()in total and this recoversabout 5.8 s of it. What makes it worth doing is where the call sits:
_save_checkpointruns synchronously inside the async
run_evolutionloop, so the time is a stallduring which no iteration is submitted and no worker result is collected.
Behaviour
The bytes on disk do not change.
json.dump(d, f)andf.write(json.dumps(d))produce identical output, and
program.to_dict()is still what gets serialized. Ichecked that rather than assuming it: I wrote a checkpoint from the same seeded
200-program population with and without the patch and compared every file by SHA-256,
once with
prompts_by_programpopulated and once with it empty. Both file sets matchand all 402 digests match.
_save_programgains aprograms_dirkeyword argument defaulting toNone. When itis
Nonethe method derives and creates the directory exactly as before, which iswhat the single-program call site in
add()depends on.python -m unittest discover testspasses, 430 tests.Headroom I deliberately left alone
Program.to_dict()isdataclasses.asdict(self), a recursive deep copy of a dictthat gets serialized and thrown away immediately. Skipping it takes the same call to
roughly 164 ms. That cost belongs to
Programrather than to the save path, and #477already proposes a change there, so this branch does not touch it.
The direction came from an automated optimization search over
openevolve/database.pyscored on the CPU cost of one
save()call, with edits confined tosaveand_save_programand every candidate checked byte-for-byte against an unmodifiedreference. The search record is at
https://dashboard.weco.ai/share/cn8uWyk3qw1XMT0mZdeMvu1fGmKCizjI (15 scored
candidates over 21 attempts, from a 238.05 ms measured baseline down to 164.46 ms). I
rebuilt this diff by hand against current
mainand re-measured it, and I offer thelink as a record of the search rather than as authorship of the patch.
The search's best candidate is not what this PR proposes. It reaches 164.46 ms partly
by dropping
program.to_dict()in favour of the live__dict__, replacingos.path.joinwith string concatenation, and disablingcheck_circularon the JSONencoder. The first collides with #477, the second breaks Windows paths, and the third
removes a safety check on user-supplied metrics and metadata. This branch keeps only
the changes that alter nothing but the cost.