Skip to content

Commit 98dbde6

Browse files
sdreyersdreyer
authored
Merge pull request #1 from ArcadeAI/cursor/it-se-7f85
2 parents 4796c41 + 00a48e2 commit 98dbde6

32 files changed

Lines changed: 5623 additions & 69 deletions

‎.dockerignore‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Git
2+
.git
3+
.gitignore
4+
5+
# GitHub
6+
.github
7+
8+
# Documentation
9+
*.md
10+
LICENSE
11+
12+
# Compiled binaries
13+
advanced_server
14+
examples/advanced_server/advanced_server
15+
16+
# IDE / editor
17+
.vscode
18+
.idea
19+
*.swp
20+
*.swo
21+
*~
22+
23+
# OS files
24+
.DS_Store
25+
Thumbs.db

‎.github/workflows/docker-publish.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: Build and Push Docker Images
2+
3+
on:
4+
push:
5+
branches: [main]
6+
tags: ["v*"]
7+
pull_request:
8+
branches: [main]
9+
10+
env:
11+
REGISTRY: ghcr.io
12+
IMAGE_PREFIX: ${{ github.repository }}
13+
14+
jobs:
15+
build-and-push:
16+
runs-on: ubuntu-latest
17+
18+
permissions:
19+
contents: read
20+
packages: write
21+
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
example:
26+
- ab_testing
27+
- advanced_server
28+
- basic_rules
29+
- content_filter
30+
- pii_redactor
31+
- user_blocking
32+
33+
steps:
34+
- name: Checkout repository
35+
uses: actions/checkout@v4
36+
37+
- name: Set up QEMU
38+
uses: docker/setup-qemu-action@v3
39+
40+
- name: Set up Docker Buildx
41+
uses: docker/setup-buildx-action@v3
42+
43+
- name: Log in to GHCR
44+
if: github.event_name != 'pull_request'
45+
uses: docker/login-action@v3
46+
with:
47+
registry: ${{ env.REGISTRY }}
48+
username: ${{ github.actor }}
49+
password: ${{ secrets.GITHUB_TOKEN }}
50+
51+
- name: Extract metadata (tags, labels)
52+
id: meta
53+
uses: docker/metadata-action@v5
54+
with:
55+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/${{ matrix.example }}
56+
tags: |
57+
type=ref,event=branch
58+
type=semver,pattern={{version}}
59+
type=semver,pattern={{major}}.{{minor}}
60+
type=semver,pattern={{major}}
61+
type=sha
62+
63+
- name: Build and push
64+
uses: docker/build-push-action@v6
65+
with:
66+
context: .
67+
file: examples/${{ matrix.example }}/Dockerfile
68+
platforms: linux/amd64,linux/arm64
69+
push: ${{ github.event_name != 'pull_request' }}
70+
tags: ${{ steps.meta.outputs.tags }}
71+
labels: ${{ steps.meta.outputs.labels }}
72+
cache-from: type=gha
73+
cache-to: type=gha,mode=max

‎README.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Hook Server Examples
2+
3+
Example webhook servers implementing the CATE (Contextual Access for Tool Execution) hook system. These servers handle access control, pre-execution validation, and post-execution filtering for tool calls.
4+
5+
## Examples
6+
7+
### Advanced Server (Full-Featured with Web UI)
8+
9+
A comprehensive server combining all features with a browser-based dashboard for configuration.
10+
11+
- **[examples/advanced_server/](examples/advanced_server/)** - Access rules, PII redaction, A/B testing, and a web UI
12+
13+
### Focused Examples (Single-Purpose, No UI)
14+
15+
Minimal servers demonstrating individual hook capabilities:
16+
17+
| Example | Description | Hook Points Used |
18+
| ------- | ----------- | ---------------- |
19+
| **[user_blocking](examples/user_blocking/)** | Block specific users from tools | Access, Pre |
20+
| **[content_filter](examples/content_filter/)** | Filter/block based on content | Access, Pre, Post |
21+
| **[pii_redactor](examples/pii_redactor/)** | Detect and redact PII in outputs | Post |
22+
| **[ab_testing](examples/ab_testing/)** | A/B and canary test tool versions | Pre |
23+
| **[basic_rules](examples/basic_rules/)** | Configurable rules for all hooks | Access, Pre, Post |
24+
25+
## Quick Start
26+
27+
```bash
28+
# Run the advanced server with the web dashboard
29+
go run ./examples/advanced_server -config ./examples/advanced_server/example-config.yaml
30+
31+
# Or run a focused example
32+
go run ./examples/pii_redactor -types "email,ssn,credit_card"
33+
go run ./examples/user_blocking -block "user1,user2"
34+
go run ./examples/content_filter -config ./examples/content_filter/example-config.yaml
35+
go run ./examples/ab_testing -config ./examples/ab_testing/example-config.yaml
36+
```
37+
38+
## Hook Points
39+
40+
These servers implement webhook endpoints that integrate with an engine's hook system:
41+
42+
| Endpoint | Purpose |
43+
| -------- | ------- |
44+
| `GET /health` | Health check |
45+
| `POST /access` | Control which tools users can see |
46+
| `POST /pre` | Validate/modify tool inputs before execution |
47+
| `POST /post` | Validate/modify tool outputs after execution |
48+
49+
## Architecture
50+
51+
```
52+
Engine Request Flow
53+
────────────────────────────────────────────────►
54+
55+
↓ ↓ ↓
56+
┌─────────┐ ┌─────────┐ ┌─────────┐
57+
│ ACCESS │ │ PRE │ │ POST │
58+
│ Hook │ │ Hook │ │ Hook │
59+
└────┬────┘ └────┬────┘ └────┬────┘
60+
│ │ │
61+
▼ ▼ ▼
62+
Can user see Can user run Filter/modify
63+
this tool? this tool? the output?
64+
```
65+
66+
## Project Structure
67+
68+
```
69+
├── examples/
70+
│ ├── advanced_server/ # Full-featured server with web UI
71+
│ ├── basic_rules/ # Configurable rules (original example)
72+
│ ├── user_blocking/ # Block specific users
73+
│ ├── content_filter/ # Content-based filtering
74+
│ ├── pii_redactor/ # PII detection and redaction
75+
│ └── ab_testing/ # A/B and canary testing
76+
├── pkg/
77+
│ └── server/ # Generated types from OpenAPI schema
78+
├── go.mod
79+
└── go.sum
80+
```

‎advanced_server‎

27.9 MB
Binary file not shown.

‎examples/ab_testing/Dockerfile‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
FROM --platform=$BUILDPLATFORM golang:1.25-alpine AS builder
2+
3+
ARG TARGETOS
4+
ARG TARGETARCH
5+
6+
WORKDIR /src
7+
8+
# Cache dependency downloads
9+
COPY go.mod go.sum ./
10+
RUN go mod download
11+
12+
# Copy shared package and example source
13+
COPY pkg/ pkg/
14+
COPY examples/ab_testing/ examples/ab_testing/
15+
16+
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
17+
go build -ldflags="-s -w" -trimpath -o /bin/server ./examples/ab_testing
18+
19+
FROM gcr.io/distroless/static-debian12
20+
21+
COPY --from=builder /bin/server /bin/server
22+
23+
EXPOSE 8080
24+
25+
ENTRYPOINT ["/bin/server"]

‎examples/ab_testing/README.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# A/B Testing Example
2+
3+
A minimal hook server that demonstrates how to **A/B test and canary-deploy tool versions** by routing tool calls to different servers.
4+
5+
## What It Shows
6+
7+
- **Pre-execution hook**: Route tool calls to different servers/versions based on experiment config
8+
- **Consistent hashing**: Same user always gets the same variant (sticky assignment)
9+
- **Weighted traffic splitting**: Control what percentage of traffic goes to each variant
10+
- **Tool registry integration**: Fetch available tools from an external API (e.g., [Arcade](https://docs.arcade.dev/en/references/api))
11+
- **Statistics tracking**: Monitor how many requests each variant receives
12+
13+
## Quick Start
14+
15+
```bash
16+
# Run with experiment config
17+
go run ./examples/ab_testing -config experiments.yaml
18+
```
19+
20+
## Config File Format
21+
22+
```yaml
23+
# Optional: external tool registry for discovering tools
24+
registry_url: "https://api.example.com"
25+
registry_key: "your-api-key"
26+
27+
experiments:
28+
# Canary deployment: 10% traffic to new version
29+
- name: "search-v2-canary"
30+
enabled: true
31+
toolkit: "Search"
32+
tool: "WebSearch"
33+
mode: canary
34+
variants:
35+
- name: "stable"
36+
weight: 90
37+
version: "1.0.0"
38+
- name: "canary"
39+
weight: 10
40+
version: "2.0.0"
41+
server_name: "search-v2"
42+
server_uri: "http://search-v2.internal:8080"
43+
server_type: "arcade"
44+
45+
# 50/50 A/B test
46+
- name: "email-provider-compare"
47+
enabled: true
48+
toolkit: "Email"
49+
tool: "*"
50+
mode: ab
51+
variants:
52+
- name: "provider-a"
53+
weight: 50
54+
- name: "provider-b"
55+
weight: 50
56+
server_name: "email-alt"
57+
server_uri: "http://email-alt.internal:8080"
58+
server_type: "arcade"
59+
```
60+
61+
## How It Works
62+
63+
### Variant Selection
64+
1. When a tool call matches an active experiment (by toolkit and tool patterns), a variant is selected
65+
2. Selection uses consistent hashing: `SHA256(user_id + ":" + experiment_name)`
66+
3. The hash is mapped to a variant based on configured weights
67+
4. The same user always gets the same variant for a given experiment
68+
69+
### Server Routing
70+
- If the selected variant has a `server_uri`, the pre-hook overrides the server routing
71+
- This allows routing to different backend servers, different tool versions, etc.
72+
- If no server override is specified, the tool executes normally (useful for tracking only)
73+
74+
### Statistics
75+
- GET `/stats` returns per-experiment, per-variant request counts
76+
- This shows the actual traffic distribution across variants
77+
78+
## Testing
79+
80+
```bash
81+
# Start with example config
82+
go run ./examples/ab_testing -config experiments.yaml &
83+
84+
# Send pre-hook requests for different users
85+
for i in $(seq 1 20); do
86+
curl -s -X POST http://localhost:8888/pre \
87+
-H "Content-Type: application/json" \
88+
-d "{
89+
\"execution_id\": \"exec-$i\",
90+
\"tool\": {\"name\": \"WebSearch\", \"toolkit\": \"Search\", \"version\": \"1.0.0\"},
91+
\"context\": {\"user_id\": \"user-$i\"},
92+
\"inputs\": {\"query\": \"test\"}
93+
}" | python3 -m json.tool
94+
echo
95+
done
96+
97+
# Check statistics
98+
curl -s http://localhost:8888/stats | python3 -m json.tool
99+
```
100+
101+
## Tool Registry Integration
102+
103+
The server can fetch available tools from an external tool registry API:
104+
105+
```bash
106+
# Configure registry URL in config, then fetch
107+
curl -s -X POST http://localhost:8888/registry/fetch | python3 -m json.tool
108+
```
109+
110+
This is useful for discovering what tools and versions are available before setting up experiments.

‎examples/ab_testing/example-config.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# A/B Testing Configuration
2+
#
3+
# Defines experiments for routing tool calls to different versions/servers.
4+
5+
# Optional: external tool registry for discovering available tools
6+
registry_url: ""
7+
registry_key: ""
8+
9+
experiments:
10+
# Canary deployment: route 10% of traffic to a new search tool version
11+
- name: "search-v2-canary"
12+
enabled: true
13+
toolkit: "Search"
14+
tool: "WebSearch"
15+
mode: canary
16+
variants:
17+
- name: "stable"
18+
weight: 90
19+
version: "1.0.0"
20+
- name: "canary"
21+
weight: 10
22+
version: "2.0.0"
23+
server_name: "search-v2"
24+
server_uri: "http://search-v2.internal:8080"
25+
server_type: "arcade"
26+
27+
# 50/50 A/B test comparing two email tool implementations
28+
- name: "email-provider-compare"
29+
enabled: true
30+
toolkit: "Email"
31+
tool: "*"
32+
mode: ab
33+
variants:
34+
- name: "provider-a"
35+
weight: 50
36+
version: "1.0.0"
37+
- name: "provider-b"
38+
weight: 50
39+
version: "1.0.0"
40+
server_name: "email-alt"
41+
server_uri: "http://email-alt.internal:8080"
42+
server_type: "arcade"

0 commit comments

Comments
 (0)