Follow-up: a7 test / skills / docs review findings

[guoqqqi]

Background

Before putting a7 into actual use, I completed a full review and local rerun of the current test suite, skills, and documentation.

Current Conclusions

• The CI baseline tests are passing.
• A local rerun can pass if it strictly follows the current CI E2E workflow.
• However, that “pass” still includes a large number of skips.
• After adding A7_GATEWAY_URL=http://192.168.10.100:19080 and HTTPBIN_URL=http://192.168.10.100:3030, many previously skipped traffic-related cases exposed real issues.
• The current skills setup has no dedicated E2E coverage and no validate-skills.
• The current overall code coverage is about 41.4%.

Main Issues

1. The test models for route / debug / stream-route / secret do not match the current EE API

Confirmed issues include:

• The route create tests still build requests based on the old model and are missing the service_id now required by EE.
• The debug trace tests still use the old route model, which causes real failures.
• The stream-route tests are inconsistent with the current schema requirements.
• The secret documentation and tests use parameter formats that do not match the current CLI.

2. Missing automation for skills

• No scripts/validate-skills.sh
• No make validate-skills
• No test/e2e/skills

3. The skills documentation is inconsistent with the current implementation and environment

Confirmed issues include:

• docs/skills.md declares skill names that do not actually exist in the repository.
• The persona documentation references commands that do not exist in the current CLI, such as health and portal.
• Several recipe documents are still written around the old route/upstream model.
• upstream create returns resource not found directly in the current EE environment.

Items That Still Need Follow-up

P0

• Rewrite the route-related E2E tests

  • Use the currently valid model: service create + route create(service_id)

• Rewrite the debug trace E2E tests

  • Base them on the current working route/service model

• Rewrite the stream-route E2E tests

  • Align them with the current schema

• Rewrite the secret E2E tests

  • Align them with the current CLI parameter format

• Rewrite auth-related scenario validation

  • Standardize on the currently supported credential create model

• Add automation for skills

  • validate-skills
  • test/e2e/skills

P1

• Systematically fix outdated model examples in the documentation

  • route
  • upstream
  • debug
  • auth
  • recipe

• Fix the nonexistent skill names and roadmap items in docs/skills.md

• Perform a cleanup pass on existing skills for actual usability in the current environment

  • Distinguish between: usable, docs need updates, depends on external systems, currently unavailable

P2

• Continue adding unit tests and gradually improve coverage
• Turn key behaviors that are currently only manually verifiable into automated tests

Suggested Deliverables

• First, complete a round of model correction

  • Unify the tests and documentation for route / credential / debug / stream-route / secret with the current EE API

• Then, add automation for skills

  • Do not pile automation on top of an outdated model

• Final outcome:

  • maintainable E2E tests
  • executable skill validation
  • documentation that matches the current implementation

[guoqqqi]

The following breakdown reflects the current state of a7, so it can be used directly for further analysis and follow-up in this issue.

Current Status

• CI baseline tests: local rerun passed
• Strictly following the current CI workflow environment variables: local E2E rerun passed
• However, that “pass” still includes a large number of skips
• After adding:
  • A7_GATEWAY_URL=http://192.168.10.100:19080
  • HTTPBIN_URL=http://192.168.10.100:3030
    many previously skipped key traffic cases exposed real issues
• There is currently no validate-skills
• There is currently no test/e2e/skills
• Total coverage: 41.4%

Main Issues Under the Strict CI Baseline

The reason the current workflow can stay green is not that all key scenarios are actually passing, but that many cases are not being executed for real.

Typical skipped cases include:

• Consumer_WithKeyAuth
• DebugTrace_*
• Most key Route_* cases
• Most key StreamRoute_* cases
• Secret_CRUD

Real Issues Exposed After Adding Gateway Addresses

After adding A7_GATEWAY_URL and HTTPBIN_URL locally, the situation changes as follows:

• TestDebugTrace_JSONOutput: real failure
• TestDebugTrace_YAMLOutput: real failure
• TestConsumer_WithKeyAuth: no longer just “missing gateway address”, but an actual credential schema mismatch
• TestRoute_TrafficForwarding: route creation is still missing service_id
• TestRoute_*: most cases are still skipped because the old route model does not match the current EE API
• TestStreamRoute_*: schema mismatch
• TestSecret_CRUD: CLI parameter format mismatch

Positive Models Confirmed to Work

The following working paths have already been verified locally:

• service create + route create(service_id) + gateway curl

  • proxy-rewrite has been verified end to end

• consumer create + credential create

  • The resource creation model for the key-auth authentication flow has been verified end to end

This shows that:

• Not all capabilities are unavailable
• But many current docs and tests are still based on the old model

Local Manual Validation Results for a7 Skills

Confirmed Working

• a7-shared
• a7-plugin-ai-prompt-decorator
• a7-plugin-ai-prompt-template
• a7-plugin-ai-proxy
• a7-plugin-basic-auth
• a7-plugin-consumer-restriction
• a7-plugin-cors
• a7-plugin-datadog
• a7-plugin-fault-injection
• a7-plugin-grpc-transcode
• a7-plugin-hmac-auth
• a7-plugin-http-logger
• a7-plugin-ip-restriction
• a7-plugin-jwt-auth
• a7-plugin-kafka-logger
• a7-plugin-key-auth
• a7-plugin-limit-count
• a7-plugin-limit-req
• a7-plugin-openid-connect
• a7-plugin-prometheus
• a7-plugin-proxy-rewrite
• a7-plugin-redirect
• a7-plugin-response-rewrite
• a7-plugin-skywalking
• a7-plugin-traffic-split
• a7-plugin-wolf-rbac
• a7-plugin-zipkin

Confirmed Issues

• a7-persona-developer
  • The docs reference a7 portal publish, but the current CLI has no such command
• a7-persona-operator
  • The docs reference a7 health, but the current CLI has no such command
• a7-plugin-ai-content-moderation
  • Querying the plugin by the current skill name returns unknown plugin
• a7-plugin-ext-plugin
  • Querying the plugin by the current skill name returns unknown plugin
• a7-plugin-serverless
  • Querying the plugin by the current skill name returns unknown plugin
• a7-recipe-api-versioning
• a7-recipe-blue-green
• a7-recipe-canary
• a7-recipe-circuit-breaker
• a7-recipe-health-check
  • These recipes all depend on upstream create
  • In the current environment, upstream create directly returns resource not found
• a7-recipe-graphql-proxy
  • The route example is still based on the old model
• a7-recipe-mtls
  • It simultaneously depends on upstream create, the old route model, and the combined secret / ssl example
• a7-recipe-multi-tenant
  • The gateway-group / consumer-group approach is still viable, but the route model is still outdated

Common Problems Across Docs and Skills

Skills docs are out of sync

• docs/skills.md claims the following exist:
  • scripts/validate-skills.sh
  • make validate-skills
  • But neither exists in the repository

Skill names and roadmap are inconsistent

• Skill / roadmap items mentioned in the docs but missing from the repository:
  • a7-recipe-gateway-group
  • a7-persona-platform-eng
  • a7-recipe-service-template
  • a7-plugin-ai-rag
  • a7-plugin-ai-token-limiter
  • a7-recipe-service-registry

Persona docs reference nonexistent commands

• health
• portal

Route / recipe docs are still based on the old model

• Most examples still directly use route + upstream
• But the currently working model should primarily shift to:
  • service create
  • route create(service_id)

Remaining Follow-up Work

P0

• Rewrite route-related E2E tests

  • Standardize on the service_id model

• Rewrite debug trace E2E tests

• Rewrite stream-route E2E tests

• Rewrite secret E2E tests

• Rewrite auth-related examples and tests

  • Standardize on credential create

• Add validate-skills

• Add test/e2e/skills

P1

• Systematically fix outdated model examples in the docs

  • route
  • upstream
  • auth
  • debug
  • recipes

• Fix nonexistent skill names and roadmap items in docs/skills.md

• Clean up all skills and classify them as:

  • usable
  • docs need updates
  • dependent on external systems
  • currently unavailable

P2

• Continue adding unit tests and improve coverage
• Gradually convert skill scenarios that have already been manually validated into automated tests