Skip to content

Conversation

@alexeyv
Copy link
Contributor

@alexeyv alexeyv commented Nov 21, 2025

Implement Workflow YAML Validation and Schema

This PR introduces comprehensive validation for all [workflow.yaml] files using Zod schema validation, ensuring consistency and catching configuration errors early in development.

🎯 Overview

Implements a robust validation system for workflow YAML files with:

  • Zod-based schema validation with comprehensive field coverage
  • Variable reference validation to catch typos in placeholders, config vars, and template vars
  • Non-fatal warning system for unknown fields (allows workflow-specific customization)
  • Comprehensive test suite with 13 test fixtures
  • CLI validator for CI/CD integration

📦 What's Included

New Files

  • tools/schema/workflow.js (450 lines) - Core Zod schema with variable reference validation
  • tools/validate-workflow-schema.js (166 lines) - CLI validator for all workflow files
  • test/test-workflow-schema.js (302 lines) - Comprehensive test runner
  • test/fixtures/workflow-schema/ - 13 test fixtures (11 invalid, 2 valid scenarios)

Key Features

  1. Schema Validation

    • Required fields: name, description, instructions
    • Kebab-case enforcement for workflow names
    • Support for all optional fields with proper typing
    • Flexible .catchall() for workflow-specific variables
  2. Variable Reference Validation

    • Path placeholders: {placeholder} - validates against 60+ known placeholders
    • Config variables: {config_source}:variable - validates against known config vars
    • Template variables: {{variable}} - validates against known template vars
    • Detects malformed syntax (unclosed braces, missing colons)
  3. Warning System

    • Non-fatal warnings for unknown fields
    • Aggregated reporting in CLI validator
    • Allows workflow-specific customization without breaking validation
  4. Testing & CI Integration

    • Integrated into npm test and npm run validate:schemas
    • Metadata-driven test expectations in YAML comments
    • Categorized test reporting with pass/fail statistics

🧹 Cleanup

Removed legacy author field from 65 workflow files across all modules (identified as LLM hallucination/legacy field not used by the system).

📊 Changes Summary

  • 85 files changed: +1,106 insertions, -92 deletions
  • Scripts updated: Added workflow validation to test:schemas and validate:schemas
  • Test coverage: 13 fixtures covering validation scenarios

✅ Testing

All validation tests pass:

npm run test:schemas        # Run test suite
npm run validate:schemas    # Validate all workflow files

@alexeyv alexeyv marked this pull request as draft November 21, 2025 20:49
@alexeyv
Copy link
Contributor Author

alexeyv commented Dec 11, 2025

This will have to be reworked to fit the new workflow format.

@alexeyv alexeyv closed this Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant