feat: chunked image push via OCI compatible push

[markphelps]

Summary

Replace Docker's monolithic ImagePush with chunked layer uploads, gated behind COG_PUSH_OCI=1. The image is exported from Docker via ImageSave, loaded lazily using go-containerregistry's tarball.ImageFromPath (layers are read on-demand from a temp file, not into memory), then each layer is pushed through the existing RegistryClient.WriteLayer chunked upload path — the same infrastructure weight artifacts already use. This bypasses request body limits that block Docker's native push for large layers.

Off by default. Set COG_PUSH_OCI=1 to enable. Falls back to Docker push on any non-fatal error (except context cancellation/timeout and auth errors), so there's zero regression risk.

What changed

Chunked image push

• ImagePusher tries OCI chunked push first when COG_PUSH_OCI=1, falls back to Docker push
• Exports image via docker.ImageSave() → tar → lazy v1.Image (via tarball.ImageFromPath) → concurrent layer upload
• shouldFallbackToDocker() blocks fallback on context cancellation/timeout and authentication errors (401/403) — auth errors won't be fixed by Docker fallback, so the original error is surfaced directly
• Added ImageSave(ctx, imageRef) to the Docker Command interface
• ImagePusher.Push() takes *ImageArtifact directly (not a raw string ref), preserving the resolved reference from the Model type

Unified push infrastructure

• Single PushProgress type replaces separate image/weight progress types
• writeLayerWithProgress() helper deduplicates progress channel boilerplate
• GetPushConcurrency() (default 5, COG_PUSH_CONCURRENCY) shared by image layers, weight pushes, and CLI progress
• BundlePusher.pushWeights() uses errgroup with concurrency limit
• NewBundlePusher takes docker.Command + registry.Client and creates both sub-pushers (image + weight) internally — unified construction

Progress rendering

• Replaced the mpb progress bar library with Docker's jsonmessage.DisplayJSONMessagesStream — the same rendering code used by docker push
• Progress output now matches docker push exactly: layer ID, status, and progress bar on each line
• Handles terminal resizing correctly: each line is erased and rewritten individually (ESC[2K + per-line cursor up/down), rather than relying on a bulk cursor-up count that desyncs when lines wrap after a resize
• Terminal width is re-queried via ioctl(TIOCGWINSZ) on every render, so progress bars adapt dynamically
• ProgressWriter adapter in pkg/docker/progress.go converts PushProgress callbacks into JSON-encoded JSONMessage structs fed to DisplayJSONMessagesStream via an io.Pipe
• Retry messages are only logged via console.Warnf in non-TTY mode — in TTY mode the progress writer handles display inline, avoiding duplicate output
• Removed mpb dependency entirely

HTTP/1.1 transport for all registry operations

• All registry operations (push, inspect, HEAD, etc.) now use HTTP/1.1 instead of allowing HTTP/2 negotiation
• Problem: Go's default http.Transport negotiates HTTP/2 via TLS ALPN. High-throughput HTTP/2 suffers from head-of-line blocking when multiplexing large blob uploads on a single connection. Additionally, certain CDN/proxy edges can send RST_STREAM INTERNAL_ERROR on large requests — killing uploads before they reach the origin.
• Fix: RegistryClient stores an http1OnlyTransport() and passes it to all remote.* calls via a remoteOptions() helper. Multiple concurrent HTTP/1.1 connections outperform a single multiplexed HTTP/2 connection for large blob uploads.
• HTTP/2 stream errors (RST_STREAM) are also recognized by isRetryableError() as a belt-and-suspenders measure, so they trigger the existing retry-from-scratch logic in WriteLayer.

Retry and resource management

• Retry jitter is now properly randomized (rand.Float64()) to avoid thundering herd when multiple clients retry simultaneously
• Chunk buffers (up to 96 MB each) are pooled via sync.Pool to reduce memory pressure when pushing multiple layers concurrently

Server-driven chunk sizing (OCI-Chunk-Min/Max-Length)

• initiateUpload() now parses OCI-Chunk-Min-Length and OCI-Chunk-Max-Length headers from the registry's 202 response
• The server-advertised maximum always takes precedence over client defaults — chunk size is set to max - 64KB margin to stay safely under the limit
• The result is clamped to be at least the server-advertised minimum
• COG_PUSH_DEFAULT_CHUNK_SIZE is only used as a fallback when the registry doesn't advertise limits
• New uploadSession struct carries location + chunk constraints from initiation through to upload

Registry config

• Extracted getDefaultChunkSize() and getMultipartThreshold() env var helpers into pkg/registry/config.go
• Default chunk size: 96 MB (multiple of 8), stays under common CDN/proxy request body limits (used only when registry doesn't advertise OCI-Chunk-Max-Length)
• Default multipart threshold: 128 MB (higher than chunk size so blobs that fit in a single chunk skip multipart entirely)

Cleanup

• Deleted pkg/oci/ package — OCI image loading inlined into ImagePusher
• Deleted tools/uploader/ — unused S3 multipart uploader (363 lines, zero imports)
• Removed Pusher interface, OCIImagePusher, pushImageWithFallback() — consolidated into single ImagePusher
• Unexported DefaultFactory → defaultFactory, NewImagePusher → newImagePusher

Environment variables

Variable	Default	Description
COG_PUSH_OCI	unset (off)	Set to 1 to enable OCI chunked image push
COG_PUSH_CONCURRENCY	5	Max concurrent layer/weight uploads
COG_PUSH_DEFAULT_CHUNK_SIZE	100663296 (96 MiB)	Fallback chunk size when registry doesn't advertise OCI-Chunk-Max-Length
COG_PUSH_MULTIPART_THRESHOLD	134217728 (128 MiB)	Blobs above this size use chunked upload; below this they upload in a single request

[markphelps]

Latest: sanitize error output and clear progress on Docker fallback

Two fixes for terminal output quality when OCI push fails and falls back to Docker push:

1. HTML response bodies no longer leak into error messages — When a registry/CDN returns an HTML error page (e.g., Cloudflare 413), transport.Error includes the entire raw body in its Error() string. Added sanitizeError() that extracts just the HTTP status code and text (e.g., HTTP 413 Request Entity Too Large) instead of dumping a full HTML page.

2. OCI progress bars are cleared before Docker fallback starts — Added OnFallback callback that closes the progressWriter before Docker push begins its own jsonmessage output. Without this, stale OCI progress lines remained on screen above Docker's progress bars.

Both fixes are covered by new tests (TestSanitizeError, TestImagePusher_OnFallback).

[bfirsh]

This is incredibly exciting.

[michaeldwan]

Strong overall, but a couple of design quirks that would be easy now to correct so the spirit of the OCI artifact & model resolver refactor isn't lost.

[markphelps]

Addressed review feedback in 6e0ed39

Worked through all unresolved comments from @mfainberg-cf and @michaeldwan. Here's what changed:

Threshold/ChunkSize inversion (pkg/registry/config.go)

Raised DefaultMultipartThreshold from 50MB → 128MB so it's above DefaultChunkSize (95MB). Blobs that fit in a single chunk now skip multipart entirely, avoiding unnecessary overhead.

HTTP/1.1 everywhere (pkg/registry/registry_client.go)

Per the discussion about HTTP/2 head-of-line blocking — all registry operations now use HTTP/1.1, not just chunked uploads. Added a transport field to RegistryClient and a remoteOptions() helper so every remote.* call uses the same HTTP/1.1 transport.

Move ProgressWriter to docker package (pkg/cli/progress.go → pkg/docker/progress.go)

Moved and exported as docker.ProgressWriter since it's built on Docker's jsonmessage rendering. CLI now imports it from there.

Errgroup in weights push (pkg/cli/weights.go)

Replaced the manual semaphore + WaitGroup + results channel with errgroup.Group + g.SetLimit(), matching the pattern already used in pusher.go. First error cancels remaining pushes.

ImagePusher.Push takes *ImageArtifact instead of string (pkg/model/image_pusher.go)

Addresses the regression concern — Push() now takes the artifact directly instead of re-parsing a string ref. Removed the now-redundant PushArtifact() method. Callers pass the already-resolved *ImageArtifact from the Model.

Unified BundlePusher construction (pkg/model/pusher.go)

NewBundlePusher now takes docker.Command + registry.Client and creates both sub-pushers (image + weight) internally. No more passing a pre-built ImagePusher from outside.

Memory concern — already handled

tarball.ImageFromPath is lazy — it indexes the tar and reads layer data on-demand from the temp file during Compressed() calls. Added a comment clarifying this. The image data is never held in memory all at once.

All tests pass, golangci-lint reports 0 issues.

[michaeldwan]

looks good, concerns addressed, ship it!