feat(ci): allow re-running tests without recreating docker images - #4676
Draft
hsuan-lun-chiang wants to merge 3 commits into
Draft
feat(ci): allow re-running tests without recreating docker images#4676hsuan-lun-chiang wants to merge 3 commits into
hsuan-lun-chiang wants to merge 3 commits into
Conversation
Implements the three user-selectable modes for running MaxText tests in Docker containers without recreating or deleting images: 1. build-all: re-run all steps including building base and runner image 2. test-only: re-run just the tests using existing image without rebuild 3. fast-rebuild: rebuild runner image over cached dependencies without --no-cache Fixes b/408067759.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
…ution Allows users running GitHub Actions test workflows via workflow_dispatch to select the execution mode from a dropdown: - build-all: rebuild all docker images and run tests - test-only: skip image building and run CI tests against existing image - fast-rebuild: rebuild docker image without --no-cache to reuse dependency cache
Allows manually triggering ci_pipeline with mode=test-only to skip package building and run tests directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Addresses b/408067759: Re-run tests without having to recreate image.
When MaxText tests fail or flake, Docker images were previously deleted or required an unconditional slow rebuild. This PR introduces three user-selectable modes for running Docker tests locally and on XPK:
--mode=build-all, default):Rebuilds both the base dependency image and the runner image before executing tests.
--mode=test-onlyor--reuse-image):Skips all image building and directly re-runs tests inside the existing Docker image without deleting it on failure.
--mode=fast-rebuildor--fast-rebuild):Skips rebuilding the slow base dependency layer and rebuilds only the runner code image (
maxtext_runner.Dockerfile) without--no-cache.Changes Included
src/dependencies/scripts/run_docker_test.sh& Python wrappersrc/dependencies/scripts/run_docker_test.py(registered asrun_maxtext_docker_testinpyproject.toml).docker_upload_runner.shandbuild_upload_colocated_python_image.shto supportFAST_REBUILD=true(andCLEANUP_IMAGE=false) to reuse Docker layer cache.reuse_image=Trueandfast_rebuild=Trueflags toWorkloadConfiginbenchmarks/maxtext_xpk_runner.py.tests/unit/run_docker_test_test.pyverifying CLI flags and XPK command generation.docs/tutorials/build_maxtext.mdwith a dedicated section on re-running tests without rebuilding images.