Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f42af45
test(commands): create extension-commands LLM playground sandbox
dhilipkumars Mar 4, 2026
1dc000b
update(tests): format LLM evaluation as an automated test runner
dhilipkumars Mar 4, 2026
c1f47b3
test(commands): map extension-commands python script with timestamps
dhilipkumars Mar 4, 2026
4d09e9b
test(commands): map extension-commands python script with timestamps
dhilipkumars Mar 4, 2026
e1e3011
test(commands): update TESTING.md to evaluate discovery, lint, and de…
dhilipkumars Mar 4, 2026
e6316fa
test(commands): simplify execution expectations and add timestamp cal…
dhilipkumars Mar 4, 2026
93d9af0
fix(tests): address copilot review comments on prompt formatting and …
dhilipkumars Mar 4, 2026
9fe81a9
fix(tests): resolve copilot PR feedback regarding extension schema st…
dhilipkumars Mar 4, 2026
1777448
feat(extensions): add core selftest utility and migrate away from man…
dhilipkumars Mar 5, 2026
48a0ea0
fix(selftest): update command name array to match spec-kit validation…
dhilipkumars Mar 5, 2026
7dc74c7
fix(selftest): wrap arguments in quotes to support multi-word extensi…
dhilipkumars Mar 5, 2026
5048406
update the command to be more meaningful
dhilipkumars Mar 9, 2026
c421117
fix: if the extension is discovery only, it should not be installable
dhilipkumars Mar 9, 2026
11b07a5
Address review comments for selftest documentation
dhilipkumars Mar 10, 2026
32b95e5
address review comments
dhilipkumars Mar 10, 2026
9e718ea
address review comments
dhilipkumars Mar 10, 2026
4247f5b
Update extensions/selftest/commands/selftest.md
mnriem Mar 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions extensions/catalog.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,21 @@
{
"schema_version": "1.0",
"updated_at": "2026-02-03T00:00:00Z",
"updated_at": "2026-03-10T00:00:00Z",
"catalog_url": "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.json",
"extensions": {}
}
"extensions": {
"selftest": {
"name": "Spec Kit Self-Test Utility",
"id": "selftest",
"version": "1.0.0",
"description": "Verifies catalog extensions by programmatically walking through the discovery, installation, and registration lifecycle.",
"author": "spec-kit-core",
"repository": "https://github.com/github/spec-kit",
"download_url": "https://github.com/github/spec-kit/releases/download/selftest-v1.0.0/selftest.zip",
"tags": [
"testing",
"core",
"utility"
]
}
}
}
69 changes: 69 additions & 0 deletions extensions/selftest/commands/selftest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
description: "Validate the lifecycle of an extension from the catalog."
---

# Extension Self-Test: `$ARGUMENTS`

This command drives a self-test simulating the developer experience with the `$ARGUMENTS` extension.

## Goal

Validate the end-to-end lifecycle (discovery, installation, registration) for the extension: `$ARGUMENTS`.
If `$ARGUMENTS` is empty, you must tell the user to provide an extension name, for example: `/speckit.selftest.extension linear`.

## Steps

### Step 1: Catalog Discovery Validation

Check if the extension exists in the Spec Kit catalog.
Execute this command and verify that it completes successfully and that the returned extension ID exactly matches `$ARGUMENTS`. If the command fails or the ID does not match `$ARGUMENTS`, fail the test.

```bash
specify extension info "$ARGUMENTS"
```

### Step 2: Simulate Installation

First, try to add the extension to the current workspace configuration directly. If the catalog provides the extension as `install_allowed: false` (discovery-only), this step is *expected* to fail.

```bash
specify extension add "$ARGUMENTS"
```

Then, simulate adding the extension by installing it from its catalog download URL, which should bypass the restriction.
Obtain the extension's `download_url` from the catalog metadata (for example, via a catalog info command or UI), then run:

```bash
specify extension add "$ARGUMENTS" --from "<download_url>"
```

### Step 3: Registration Verification

Once the `add` command completes, verify the installation by checking the project configuration.
Use terminal tools (like `cat`) to verify that the following file contains a record for `$ARGUMENTS`.

```bash
cat .specify/extensions/.registry/$ARGUMENTS.json
```

### Step 4: Verification Report

Analyze the standard output of the three steps.
Generate a terminal-style test output format detailing the results of discovery, installation, and registration. Return this directly to the user.

Example output format:
```text
============================= test session starts ==============================
collected 3 items

test_selftest_discovery.py::test_catalog_search [PASS/FAIL]
Details: [Provide execution result of specify extension search]

test_selftest_installation.py::test_extension_add [PASS/FAIL]
Details: [Provide execution result of specify extension add]

test_selftest_registration.py::test_config_verification [PASS/FAIL]
Details: [Provide execution result of registry record verification]

============================== [X] passed in ... ==============================
```
16 changes: 16 additions & 0 deletions extensions/selftest/extension.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
schema_version: "1.0"
extension:
id: selftest
name: Spec Kit Self-Test Utility
version: 1.0.0
description: Verifies catalog extensions by programmatically walking through the discovery, installation, and registration lifecycle.
author: spec-kit-core
repository: https://github.com/github/spec-kit
license: MIT
requires:
speckit_version: ">=0.2.0"
provides:
commands:
- name: speckit.selftest.extension
file: commands/selftest.md
description: Validate the lifecycle of an extension from the catalog.
Loading