Skip to content

Commit 85192ff

Browse files
gmazoyergmazoyer
committed
Add plan to implement INFP-504 (#885)
1 parent ced2533 commit 85192ff

7 files changed

Lines changed: 739 additions & 5 deletions

File tree

‎dev/specs/infp-504-artifact-composition/contracts/filter-interfaces.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Filter Interface Contracts
2+
3+
**Feature**: INFP-504 | **Date**: 2026-03-20
4+
5+
## Jinja2 Filter Signatures
6+
7+
### artifact_content
8+
9+
```python
10+
async def artifact_content(storage_id: str) -> str
11+
```
12+
13+
| Input | Output | Error |
14+
| ----- | ------ | ----- |
15+
| Valid storage_id string | Raw artifact content (text) | — |
16+
| `None` | — | `JinjaFilterError("artifact_content", "storage_id is null", hint="...")` |
17+
| `""` (empty) | — | `JinjaFilterError("artifact_content", "storage_id is empty", hint="...")` |
18+
| Non-existent storage_id | — | `JinjaFilterError("artifact_content", "content not found: {id}")` |
19+
| Permission denied (401/403) | — | `JinjaFilterError("artifact_content", "permission denied for storage_id: {id}")` |
20+
| No client provided | — | `JinjaFilterError("artifact_content", "requires InfrahubClient", hint="pass client via Jinja2Template(client=...)")` |
21+
22+
**Validation**: Blocked in `CORE` context. Allowed in `WORKER` context.
23+
24+
### file_object_content
25+
26+
```python
27+
async def file_object_content(storage_id: str) -> str
28+
```
29+
30+
| Input | Output | Error |
31+
| ----- | ------ | ----- |
32+
| Valid storage_id (text file) | Raw file content (text) | — |
33+
| Valid storage_id (binary file) | — | `JinjaFilterError("file_object_content", "binary content not supported for storage_id: {id}")` |
34+
| `None` | — | `JinjaFilterError("file_object_content", "storage_id is null", hint="...")` |
35+
| `""` (empty) | — | `JinjaFilterError("file_object_content", "storage_id is empty", hint="...")` |
36+
| Non-existent storage_id | — | `JinjaFilterError("file_object_content", "content not found: {id}")` |
37+
| Permission denied (401/403) | — | `JinjaFilterError("file_object_content", "permission denied for storage_id: {id}")` |
38+
| No client provided | — | `JinjaFilterError("file_object_content", "requires InfrahubClient", hint="pass client via Jinja2Template(client=...)")` |
39+
40+
**Validation**: Blocked in `CORE` context. Allowed in `WORKER` context.
41+
42+
### from_json
43+
44+
```python
45+
def from_json(value: str) -> dict | list
46+
```
47+
48+
| Input | Output | Error |
49+
| ----- | ------ | ----- |
50+
| Valid JSON string | Parsed dict or list | — |
51+
| `""` (empty) | `{}` | — |
52+
| Malformed JSON | — | `JinjaFilterError("from_json", "invalid JSON: {error_detail}")` |
53+
54+
**Validation**: Allowed in all contexts (`ALL`).
55+
56+
### from_yaml
57+
58+
```python
59+
def from_yaml(value: str) -> dict | list
60+
```
61+
62+
| Input | Output | Error |
63+
| ----- | ------ | ----- |
64+
| Valid YAML string | Parsed dict, list, or scalar | — |
65+
| `""` (empty) | `{}` | — |
66+
| Malformed YAML | — | `JinjaFilterError("from_yaml", "invalid YAML: {error_detail}")` |
67+
68+
**Validation**: Allowed in all contexts (`ALL`).
69+
70+
## ObjectStore API Contract
71+
72+
### GET /api/storage/object/{identifier} (existing)
73+
74+
Used by `artifact_content`. Returns plain text content.
75+
76+
### GET /api/files/by-storage-id/{storage_id} (new)
77+
78+
Used by `file_object_content`. Returns file content with appropriate content-type header.
79+
80+
**Accepted content-types** (text-based):
81+
82+
- `text/*`
83+
- `application/json`
84+
- `application/yaml`
85+
- `application/x-yaml`
86+
87+
**Rejected**: All other content-types → `JinjaFilterError` with binary content message.
88+
89+
## Validation Contract
90+
91+
### validate() method
92+
93+
```python
94+
def validate(
95+
self,
96+
restricted: bool = True,
97+
context: ExecutionContext | None = None,
98+
) -> None
99+
```
100+
101+
| Context | Trusted filters | Worker filters | Untrusted filters |
102+
| ------- | :-: | :-: | :-: |
103+
| `CORE` | allowed | blocked | blocked |
104+
| `WORKER` | allowed | allowed | blocked |
105+
| `LOCAL` | allowed | allowed | allowed |
106+
107+
**Backward compat**: `restricted=True` → `CORE`, `restricted=False` → `LOCAL`.

‎dev/specs/infp-504-artifact-composition/data-model.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# Data Model: Artifact Content Composition
2+
3+
**Feature**: INFP-504 | **Date**: 2026-03-20
4+
5+
## New Entities
6+
7+
### ExecutionContext (Flag enum)
8+
9+
**Location**: `infrahub_sdk/template/filters.py`
10+
11+
```python
12+
class ExecutionContext(Flag):
13+
CORE = auto() # API server computed attributes — most restrictive
14+
WORKER = auto() # Prefect background workers
15+
LOCAL = auto() # Local CLI / unrestricted rendering
16+
ALL = CORE | WORKER | LOCAL
17+
```
18+
19+
**Semantics**: Represents where template code executes. A filter's `allowed_contexts` flags are an allowlist — fewer flags means less trusted.
20+
21+
### FilterDefinition (modified)
22+
23+
**Location**: `infrahub_sdk/template/filters.py`
24+
25+
```python
26+
@dataclass
27+
class FilterDefinition:
28+
name: str
29+
allowed_contexts: ExecutionContext
30+
source: str
31+
32+
@property
33+
def trusted(self) -> bool:
34+
"""Backward compatibility: trusted means allowed in all contexts."""
35+
return self.allowed_contexts == ExecutionContext.ALL
36+
```
37+
38+
**Migration**:
39+
40+
| Current | New |
41+
| ------- | --- |
42+
| `FilterDefinition("abs", trusted=True, source="jinja2")` | `FilterDefinition("abs", allowed_contexts=ExecutionContext.ALL, source="jinja2")` |
43+
| `FilterDefinition("safe", trusted=False, source="jinja2")` | `FilterDefinition("safe", allowed_contexts=ExecutionContext.LOCAL, source="jinja2")` |
44+
45+
### JinjaFilterError (new exception)
46+
47+
**Location**: `infrahub_sdk/template/exceptions.py`
48+
49+
```python
50+
class JinjaFilterError(JinjaTemplateError):
51+
def __init__(self, filter_name: str, message: str, hint: str | None = None) -> None:
52+
self.filter_name = filter_name
53+
self.hint = hint
54+
full_message = f"Filter '{filter_name}': {message}"
55+
if hint:
56+
full_message += f" — {hint}"
57+
super().__init__(full_message)
58+
```
59+
60+
**Inheritance**: `Error` → `JinjaTemplateError` → `JinjaFilterError`
61+
62+
### InfrahubFilters (new class)
63+
64+
**Location**: `infrahub_sdk/template/infrahub_filters.py` (new file)
65+
66+
```python
67+
class InfrahubFilters:
68+
def __init__(self, client: InfrahubClient) -> None:
69+
self.client = client
70+
71+
async def artifact_content(self, storage_id: str) -> str:
72+
"""Retrieve artifact content by storage_id."""
73+
...
74+
75+
async def file_object_content(self, storage_id: str) -> str:
76+
"""Retrieve file object content by storage_id."""
77+
...
78+
```
79+
80+
**Key design decisions**:
81+
82+
- Methods are `async` — Jinja2's `auto_await` handles them in async rendering mode
83+
- Holds an `InfrahubClient` (async only), not `InfrahubClientSync`
84+
- Each method validates inputs and catches `AuthenticationError` to wrap in `JinjaFilterError`
85+
86+
## Modified Entities
87+
88+
### Jinja2Template (modified constructor)
89+
90+
**Location**: `infrahub_sdk/template/__init__.py`
91+
92+
```python
93+
def __init__(
94+
self,
95+
template: str | Path,
96+
template_directory: Path | None = None,
97+
filters: dict[str, Callable] | None = None,
98+
client: InfrahubClient | None = None, # NEW
99+
) -> None:
100+
```
101+
102+
**Changes**:
103+
104+
- New optional `client` parameter
105+
- When `client` provided: instantiate `InfrahubFilters`, register `artifact_content` and `file_object_content`
106+
- Always register `from_json` and `from_yaml` (no client needed)
107+
- File-based environment must add `enable_async=True` for async filter support
108+
109+
### Jinja2Template.validate() (modified signature)
110+
111+
```python
112+
def validate(self, restricted: bool = True, context: ExecutionContext | None = None) -> None:
113+
```
114+
115+
**Changes**:
116+
117+
- New optional `context` parameter (takes precedence over `restricted` when provided)
118+
- Backward compat: `restricted=True` → `ExecutionContext.CORE`, `restricted=False` → `ExecutionContext.LOCAL`
119+
- Validation logic: filter allowed if `filter.allowed_contexts & context` is truthy
120+
121+
### ObjectStore (new method)
122+
123+
**Location**: `infrahub_sdk/object_store.py`
124+
125+
```python
126+
async def get_file_by_storage_id(self, storage_id: str, tracker: str | None = None) -> str:
127+
"""Retrieve file object content by storage_id.
128+
129+
Raises error if content-type is not text-based.
130+
"""
131+
...
132+
```
133+
134+
**API endpoint**: `GET /api/files/by-storage-id/{storage_id}`
135+
136+
**Content-type check**: Allow `text/*`, `application/json`, `application/yaml`, `application/x-yaml`. Reject all others.
137+
138+
## New Filter Registrations
139+
140+
```python
141+
# In AVAILABLE_FILTERS:
142+
143+
# Infrahub client-dependent filters (worker context only)
144+
FilterDefinition("artifact_content", allowed_contexts=ExecutionContext.WORKER, source="infrahub"),
145+
FilterDefinition("file_object_content", allowed_contexts=ExecutionContext.WORKER, source="infrahub"),
146+
147+
# Parsing filters (trusted, all contexts)
148+
FilterDefinition("from_json", allowed_contexts=ExecutionContext.ALL, source="infrahub"),
149+
FilterDefinition("from_yaml", allowed_contexts=ExecutionContext.ALL, source="infrahub"),
150+
```
151+
152+
## Relationships
153+
154+
```text
155+
Jinja2Template
156+
├── has-a → InfrahubFilters (when client provided)
157+
├── uses → FilterDefinition registry (for validation)
158+
└── uses → ExecutionContext (for context-aware validation)
159+
160+
InfrahubFilters
161+
├── has-a → InfrahubClient
162+
└── uses → ObjectStore (for content retrieval)
163+
164+
JinjaFilterError
165+
└── extends → JinjaTemplateError → Error
166+
```

‎dev/specs/infp-504-artifact-composition/plan.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Implementation Plan: Artifact Content Composition
2+
3+
**Branch**: `infp-504-artifact-composition` | **Date**: 2026-03-20 | **Spec**: [spec.md](spec.md)
4+
**Jira**: INFP-504 | **Epic**: IFC-2275
5+
6+
## Summary
7+
8+
Enable Jinja2 templates to reference and inline rendered content from other artifacts and file objects via new filters (`artifact_content`, `file_object_content`, `from_json`, `from_yaml`). Requires evolving the filter trust model from a binary boolean to a flag-based execution context system, creating a new `InfrahubFilters` class to hold client-dependent filter logic, and extending `Jinja2Template` with an optional client parameter.
9+
10+
## Technical Context
11+
12+
**Language/Version**: Python 3.10-3.13
13+
**Primary Dependencies**: jinja2, httpx, pydantic >=2.0, PyYAML (already available via netutils)
14+
**Storage**: Infrahub object store (REST API)
15+
**Testing**: pytest (`uv run pytest tests/unit/`)
16+
**Target Platform**: SDK library consumed by Prefect workers, CLI, and API server
17+
**Project Type**: Single Python package
18+
**Constraints**: No new external dependencies. Must maintain async/sync dual pattern. Must not break existing filter behavior.
19+
20+
## Key Technical Decisions
21+
22+
### 1. Async Filters via Jinja2 native support (R-001)
23+
24+
The `SandboxedEnvironment` already uses `enable_async=True`. Jinja2's `auto_await` automatically awaits filter return values during `render_async()`. The new content-fetching filters can be `async def` — no bridging needed.
25+
26+
**Required change**: Add `enable_async=True` to the file-based environment (`_get_file_based_environment()`) so async filters work for file-based templates too.
27+
28+
### 2. Flag-based trust model (R-004)
29+
30+
Replace `FilterDefinition.trusted: bool` with `allowed_contexts: ExecutionContext` using Python's `Flag` enum. Three contexts: `CORE` (most restrictive), `WORKER`, `LOCAL` (least restrictive). A backward-compatible `trusted` property preserves existing API.
31+
32+
### 3. Content-type checking for file objects (R-003)
33+
34+
New `ObjectStore.get_file_by_storage_id()` method checks response `content-type` header. Text-based types are allowed; binary types are rejected with a descriptive error.
35+
36+
## Project Structure
37+
38+
### Documentation (this feature)
39+
40+
```text
41+
dev/specs/infp-504-artifact-composition/
42+
├── spec.md # Feature specification
43+
├── plan.md # This file
44+
├── research.md # Phase 0 research findings
45+
├── data-model.md # Entity definitions
46+
├── quickstart.md # Usage examples
47+
├── contracts/
48+
│ └── filter-interfaces.md # Filter I/O contracts
49+
└── checklists/
50+
└── requirements.md # Quality checklist
51+
```
52+
53+
### Source Code (files to create or modify)
54+
55+
```text
56+
infrahub_sdk/
57+
├── template/
58+
│ ├── __init__.py # MODIFY: Jinja2Template (client param, validate context)
59+
│ ├── filters.py # MODIFY: ExecutionContext enum, FilterDefinition migration
60+
│ ├── exceptions.py # MODIFY: Add JinjaFilterError
61+
│ └── infrahub_filters.py # CREATE: InfrahubFilters class
62+
├── object_store.py # MODIFY: Add get_file_by_storage_id()
63+
```
64+
65+
```text
66+
tests/unit/
67+
├── template/
68+
│ ├── test_filters.py # MODIFY: Tests for new filters and trust model
69+
│ └── test_infrahub_filters.py # CREATE: Tests for InfrahubFilters
70+
```
71+
72+
## Implementation Order
73+
74+
The 13 Jira tasks under IFC-2275 follow this dependency graph:
75+
76+
```text
77+
Phase 1 (Foundation — no dependencies, can be parallel):
78+
IFC-2367: JinjaFilterError exception
79+
IFC-2368: Flag-based trust model (ExecutionContext + FilterDefinition migration)
80+
IFC-2373: ObjectStore.get_file_by_storage_id()
81+
82+
Phase 2 (Filters — depend on Phase 1):
83+
IFC-2369: from_json filter (depends on IFC-2367)
84+
IFC-2370: from_yaml filter (depends on IFC-2367)
85+
IFC-2371: InfrahubFilters class (depends on IFC-2367)
86+
87+
Phase 3 (Content filters — depend on Phase 2):
88+
IFC-2372: artifact_content filter (depends on IFC-2371)
89+
IFC-2374: file_object_content filter (depends on IFC-2371, IFC-2373)
90+
91+
Phase 4 (Integration — depend on Phase 3):
92+
IFC-2375: Jinja2Template client param + wiring (depends on IFC-2368, IFC-2371, IFC-2372)
93+
IFC-2376: Filter registration with correct contexts (depends on IFC-2368, IFC-2369, IFC-2370, IFC-2372, IFC-2374)
94+
95+
Phase 5 (Documentation + Server — depend on Phase 4):
96+
IFC-2377: Documentation (depends on IFC-2376)
97+
IFC-2378: integrator.py threading [Infrahub server] (depends on IFC-2375)
98+
IFC-2379: Schema validation [Infrahub server] (depends on IFC-2368)
99+
```
100+
101+
## Risk Register
102+
103+
| Risk | Likelihood | Impact | Mitigation |
104+
| ---- | --------- | ------ | ---------- |
105+
| Jinja2 `auto_await` doesn't work as expected for filters | Low | High | Verify with a minimal test before building on the assumption. Fallback: sync wrapper with thread executor. |
106+
| File-based environment breaks with `enable_async=True` | Low | Medium | File-based env change is isolated and testable. Existing tests will catch regressions. |
107+
| ObjectStore API returns incorrect content-type for file objects | Medium | Low | Already flagged by @wvandeun. The filter will use best-effort content-type checking; can be refined when API is fixed. |
108+
| `validate()` backward compat breaks existing callers | Low | High | Keep `restricted` param with deprecation path. Test all existing call sites. |

0 commit comments

Comments
 (0)