fix(compute): recover Error-phase sandboxes on gateway startup

[r3v5]

Summary

• After a Podman/Docker machine restart, sandbox containers exit with SIGTERM (code 143) but remain on disk. The gateway marked them Error and on next startup the resume sweep skipped Error-phase sandboxes entirely — leaving them stuck until the user deleted and recreated them, losing state.
• resume_persisted_sandboxes() now attempts Error-phase sandboxes: if the container still exists it is restarted and the sandbox transitions back to Provisioning → Ready; if the container is gone or the driver fails, the sandbox stays in Error without overwriting the original error reason.
• Adds resume_sandbox() to the Podman driver and wires StartupResume for PodmanComputeDriver so the resume sweep runs on Podman-backed gateways (previously Docker-only).

Related Issue

Closes #2179

Changes

• crates/openshell-driver-podman/src/driver.rs — Added resume_sandbox() method that inspects container state and calls start_container if the container exists but is not running.
• crates/openshell-server/src/compute/mod.rs:
  • Modified phase filter in resume_persisted_sandboxes() to only skip Deleting (not Error)
  • Added clear_sandbox_error() method (mirrors mark_sandbox_error()) to reset phase to Provisioning and set Ready condition to "Resumed"
  • Guarded Ok(false) and Err arms to avoid overwriting existing error state on already-Error sandboxes
  • Implemented StartupResume for PodmanComputeDriver and wired it into new_podman()
  • Added recovered counter to summary log line for observability
  • Removed unused sandbox_phase_should_be_running() function
  • Updated doc comments to reflect new behavior

Testing

Unit tests (3 new + 1 updated)

• resume_persisted_sandboxes_recovers_error_phase_when_container_exists — Ok(true) → phase becomes Provisioning, Ready condition reason = "Resumed"
• resume_persisted_sandboxes_leaves_error_when_container_missing — Ok(false) → phase stays Error, no overwrite
• resume_persisted_sandboxes_leaves_error_when_resume_fails — Err → phase stays Error, no overwrite
• Updated resume_persisted_sandboxes_resumes_running_phases to expect Error-phase sandbox in called_ids

All 1032 tests pass across openshell-server and openshell-driver-podman. Clippy clean.

Manual verification

Full end-to-end reproduction of issue #2179:

1. Created sandbox with Vertex AI provider: openshell sandbox create --name my-podman-sandbox --provider vertex-prod
2. Verified sandbox Ready and container Up (healthy)
3. Stopped Podman machine: podman machine stop
4. Restarted Podman machine: podman machine start
5. Verified container Exited (143) and sandbox stuck in Error (before fix — screenshot 1)
6. Restarted gateway with fix — logs show Resumed sandbox ... phase=Error recovered=true and Sandbox resume sweep complete resumed=1 recovered=1 (screenshot 2)
7. Verified sandbox recovered to Ready and container Up (healthy) (screenshot 2)
8. Verified sandbox usable via openshell term TUI (screenshot 3)

Screenshot 1 — Before fix: sandbox stuck in Error after Podman machine restart

Screenshot 2 — After fix: gateway restart recovers sandbox to Ready

Screenshot 3 — Sandbox usable in OpenShell TUI after recovery

Checklist

• Follows Conventional Commits format
• Signed off (DCO)
• Unit tests added and passing
• Pre-commit checks passing
• Manual end-to-end verification complete
• No proto changes, no new gRPC RPCs, no CLI changes

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[r3v5]

Manual Verification Screenshots

Screenshot 1 — Before fix: sandbox stuck in Error after Podman machine restart (exit code 143)

Screenshot 2 — After fix: gateway restart recovers sandbox, logs show recovered=true, sandbox Ready

Screenshot 3 — Sandbox usable in OpenShell TUI after recovery

[r3v5]

I have read the DCO document and I hereby sign the DCO.

[r3v5]

recheck

[r3v5]

PR Review — Claude Code (Opus 4.6)

Overview

After Podman/Docker machine restart, sandbox containers exit with SIGTERM but remain on disk. Gateway previously skipped Error-phase sandboxes during resume sweep, leaving them stuck. This PR makes the resume sweep attempt Error-phase sandboxes — restarting containers that still exist, leaving Error untouched if the container is gone or the driver fails. Also wires StartupResume for Podman (was Docker-only).

Key Design Decisions

• Include Error in resume sweep, only skip Deleting (crates/openshell-server/src/compute/mod.rs:787) — Replaces sandbox_phase_should_be_running() filter with simple phase == Deleting check. Error-phase sandboxes now get a recovery attempt instead of being permanently stuck.

• Guard against overwriting existing error state (mod.rs:810-828) — When an already-Error sandbox fails resume (Ok(false) or Err), the original error reason is preserved. Only Ok(true) clears error state. Prevents losing diagnostic info.

• clear_sandbox_error() resets to Provisioning, not Ready (mod.rs:895-930) — Recovered sandboxes go to Provisioning with Ready condition reason "Resumed". Watch loop then drives them to Ready naturally, same path as fresh sandboxes.

• Podman resume_sandbox() uses inspect-then-start (crates/openshell-driver-podman/src/driver.rs:695-712) — Inspects container first; if running returns Ok(true) without restart. NotFound mapped to Ok(false) at both inspect and start points for race safety.

Notable Code

crates/openshell-driver-podman/src/driver.rs:695:

pub async fn resume_sandbox(&self, sandbox_name: &str) -> Result<bool, ComputeDriverError> {
    let name = container::container_name(sandbox_name);
    let inspect = match self.client.inspect_container(&name).await {
        Ok(i) => i,
        Err(PodmanApiError::NotFound(_)) => return Ok(false),
        Err(e) => return Err(ComputeDriverError::from(e)),
    };
    if inspect.state.running {
        return Ok(true);
    }
    match self.client.start_container(&name).await {
        Ok(()) => Ok(true),
        Err(PodmanApiError::NotFound(_)) => Ok(false),
        Err(e) => Err(ComputeDriverError::from(e)),
    }
}

Potential Concerns

• TOCTOU between inspect and start — Container could be removed between inspect_container and start_container. Handled by catching NotFound on start, so no actual bug, but worth noting the race is accounted for.
• update_message_cas with generation 0 in clear_sandbox_error() (mod.rs:904) — Passing 0 as CAS generation means no optimistic concurrency check. Safe here since this runs at startup before watchers spawn (per doc comment), but fragile if ever called later. Matches existing mark_sandbox_error() pattern though.

Verdict

Solid fix. Tests cover all three outcomes (container exists, gone, driver error). Manual E2E verification thorough. Clean removal of now-unnecessary sandbox_phase_should_be_running(). No blocking concerns.

[r3v5]

Unused anymore after the changes.