Rewrite API version update guide as tested, validated Claude skill#3433
Rewrite API version update guide as tested, validated Claude skill#3433
Conversation
|
Oxygen deployed a preview of your
Learn more about Hydrogen's GitHub integration. |
Replace 1618-line command with 304-line skill (81% reduction). The old guide had issues with LLM hallucinations (suggesting fixes, analyzing errors), duplicate sections (build steps appeared 13+ times), and overly complex templates. The new skill: - Adds explicit anti-hallucination directive at the top - Consolidates build steps to appear exactly once - Uses minimal child issue template to prevent AI interpretation - Clarifies "8 Tracking Phases" vs "9 Implementation Steps" - Adds handoff note for phases 3-8 requiring manual execution - Strengthens checkpoints: "report EXACT error... Do NOT diagnose"
kdaviduik
force-pushed
the
2026-01-update
branch
from
f6faf9a to
2528d9e
Compare
kdaviduik
changed the title
Confirmed all GitHub API calls work via live test (issues #1084, #1085, #1086 in Shopify/developer-tools-team). Changes: - Replace all HEREDOC patterns with Write tool + --body-file (HEREDOC blocked in sandbox) - Add Status field mutation alongside Project field in all three code paths (Step 7a, Step 8d, setup_phase_child_issue function) - Add Status verification to all three code paths and to Checkpoint 3 - Document 'child' label pre-exists (no creation needed) - Add Success Criteria Checklist section - Add new project constants: Status field ID (PVTSSF_lADNH5XOABDyGM4AnCva) and Todo-prioritized option ID (f75ad846) - Add Lesson 6: HEREDOC blocked in sandbox - Add Lesson 7: GraphQL variable interpolation with '"ENV_VAR"' pattern - Fix literal <PARENT_NUM> placeholders in Step 8e GraphQL queries (bug: they weren't expanding) - Save PARENT_NUM to /tmp/claude/parent_num.txt for cross-step recovery
kdaviduik
changed the title
| 4. **Manual testing** - Test changes in a real Hydrogen project | ||
| 5. **Release versions** - Publish packages to npm | ||
| 6. **Add post in docs/changelog.json** - Enable `h2 upgrade` command | ||
| 7. **Update Shopify dev API reference docs** - Sync external documentation |
There was a problem hiding this comment.
note: theoretically the dev docs update step should be automated, but I think this skill is more effective if it creates a comprehensive list of all the things that must happen, to ensure nothing slips through the cracks
|
These findings are in files not modified by this PR and cannot be posted as inline comments.
In Step 7a, verification reads Impact: Reliability hazard for the “validated skill” claim; the skill may exit 1 incorrectly or pass incorrectly, leaving issues in the wrong project/status. |
|
🤖 Code Review · #projects-dev-ai for questions ✅ Complete - No issues 📋 History✅ No issues → ✅ No issues → ✅ No issues |
…ueries
All 8 verification queries used nodes[0] to read field values, assuming the first project item belongs to project #4613. This assumption is incorrect — an issue can belong to multiple projects and GraphQL ordering is not stable.
Fix: Add project { number } to the nodes fragment in all verification queries, then filter using nodes[] | select(.project.number == 4613) before reading .fieldValueByName.name. This mirrors the correct pattern already used in the item-ID lookup queries.
|
These findings are in files not modified by this PR and cannot be posted as inline comments.
The skill assumes Evidence: PARENT_ISSUE_URL=$(gh issue create ... --body-file /tmp/claude/parent-body-initial.md)
PARENT_NUM=$(echo "$PARENT_ISSUE_URL" | grep -oE '[0-9]+$')
The skill assumes
In Step 7a, Evidence: PARENT_ITEM_ID=$(gh api graphql ... --jq '... | .id')
# later used as itemId: "$PARENT_ITEM_ID" |
…parse Three brittle silent-failure paths fixed: 1. PARENT_NUM: guard after grep-oE parse of gh issue create output 2. ISSUE_NUM: guard in Step 8c and all 6 phase child blocks (replace_all) 3. PARENT_ITEM_ID: guard in Step 7a before GraphQL mutations Without these guards, an unexpected gh output format or a race on project item creation propagates an empty variable silently through downstream GraphQL mutations, jq filters, and sub-issue linking — producing misleading errors or silent incorrect behavior.
fredericoo
left a comment
fredericoo
left a comment
There was a problem hiding this comment.
is this PR still valid? if so, here are some comments
there are some merge conflicts because we already moved it all to skills
|
|
||
| ```bash | ||
| ISSUE_URL=$(gh issue create --repo Shopify/developer-tools-team \ | ||
| --title "Release <VERSION_CALVER> versions" \ |
There was a problem hiding this comment.
non-blocking: <VERSION_CALVER> isn't defined anywhere in the skill - every other reference uses <VERSION> (e.g., 2026-01). If CalVer is intentionally different (e.g., 2026.1.0), let's define it in Step 1 alongside <VERSION>. Otherwise, should this just be <VERSION>?
| Then: | ||
|
|
||
| ```bash | ||
| git add -A |
There was a problem hiding this comment.
non-blocking: git add -A stages everything including any temp files, .env files, or IDE state that might be in the working directory. For a skill that an agent executes semi-autonomously, let's use explicit paths instead:
| git add -A | |
| git add packages/ templates/ .changeset/ |
| grep -o 'shopify.dev/changelog' /tmp/claude/api_changes.json | head -1 && echo "✓ URLs transformed" || echo "✗ URL transformation failed" | ||
| ``` | ||
|
|
||
| ### 8c: For Each Changelog Entry |
| grep "SF_API_VERSION" packages/hydrogen-react/codegen.ts | ||
|
|
||
| # Create feature branch | ||
| git checkout -b <VERSION>-sfapi-caapi-update |
There was a problem hiding this comment.
nit: let's include origin/main as the base branch to avoid picking up unrelated local changes (happened to me):
| git checkout -b <VERSION>-sfapi-caapi-update | |
| git checkout -b <VERSION>-sfapi-caapi-update origin/main |
| # Add parent issue to project | ||
| gh project item-add 4613 --owner Shopify --url "$PARENT_ISSUE_URL" | ||
| sleep 2 | ||
|
|
There was a problem hiding this comment.
question: was the sleep 2 determined empirically during the live test? Lesson 4 documents it as part of the required sequence. For a quarterly process this is pragmatic - just checking if there's a known failure mode when GitHub's project API is slower than usual.
2 participants
WHY are these changes introduced?
The original
.claude/commands/apis-version-update.md(1618 lines) had two categories of problems:Structural issues (from previous session):
Untested API patterns (discovered this session):
cat <<'EOF') is blocked in the Claude Code sandbox — issue creation would silently fail<PARENT_NUM>literal placeholders in Step 8e GraphQL queries — these were passed literally, not as shell variables, causing sub-issue linking to failWHAT is this pull request doing?
Replaces the 1618-line command with a ~900-line skill, validated via live test against real GitHub issues (#1084, #1085, #1086 in
Shopify/developer-tools-team):.claude/commands/apis-version-update.md.claude/skills/apis-version-update/SKILL.mdWritetool +--body-file<PARENT_NUM>placeholder'""'expansionAll 9 success criteria were confirmed passing via live API test before updating the skill.
HOW to test your changes?
/apis-version-updateChecklist