Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,27 @@ both explicitly and may use pi's orchestration primitives directly.
`@onkernel/loop` is one package with two entry points and three source trees:

- `.` (`src/core/`) is the framework-neutral core: canonical actions, the tool
declarations namespace, the catalog compiler, the tool menu, the tool manager,
and Kernel-browser execution (translator, CDP executor, execution resources).
Catalog compilation is declaration-only and deterministic. Its coupling to pi
is type-level — `Api`, `Model`, `Tool`, `AgentTool` — except for the model
resolution and provider modules it still reaches into, which the next split
moves behind an interface.
- `./pi` (`src/pi/`) is the pi binding: `attach()`/`compile()`, model
resolution, transport derivation, the provider adapters, provider retry, and
header composition.
declarations namespace, the catalog compiler, the tool menu, and
Kernel-browser execution (translator, CDP executor, execution resources).
Catalog compilation is declaration-only and deterministic. The core imports
nothing from pi — declarations are `LoopToolDeclaration`, executables are
`LoopExecutableTool` with an `(input, signal)` contract, models are the
neutral `LoopCatalogModel` view, and schemas come from `typebox` directly.
Per-model availability (capability quirks, native-surface tables) and
provider request preparation stay on the pi side: the binding hands the
compiler `LoopModelFacts` and `model-preparation` transforms as inputs, and
core only orders and validates what it is given.
`test/core-boundary.test.ts` fails the unit suite on any `src/core` import
that is not core-relative or an allowlisted neutral dependency — including
pi packages and this package's own `@onkernel/loop/pi` subpath.
- `./pi` (`src/pi/`) is the pi binding: `attach()`/`compile()`, the tool
manager that joins compiled catalogs to executable pi `AgentTool`s, model
resolution and availability facts (`compileLoopToolCatalog`/`loopToolMenu`
accept provider-qualified refs here and supply `LoopModelFacts`,
`modelSupportsDeferredTools` interprets pi compat flags, and the published
`loop` namespace composes `loop.providers.anthropic.supports` over core's
declarations), transport derivation, the provider adapters, provider retry,
and header composition.
- `src/pi-extension/` contributes these tools to a pi session that pi itself
owns. It is the one consumer that uses neither `attach()` nor the harness: pi
owns the model collection and the agent loop, so the extension takes the two
Expand Down
3 changes: 2 additions & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions packages/loop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Two entry points:

| import | what it is |
| --- | --- |
| `@onkernel/loop` | The framework-neutral core: canonical actions, the tool namespace, catalog compilation, the tool menu, and Kernel-browser execution. |
| `@onkernel/loop` | The framework-neutral core: canonical actions, the tool namespace, catalog compilation, the tool menu, and Kernel-browser execution. Core declarations (`LoopToolDeclaration`) and executables (`LoopExecutableTool`) import nothing from pi — schemas come from `typebox` directly — and a unit test enforces the boundary. |
| `@onkernel/loop/pi` | The pi binding: `attach()`, model resolution, transport derivation, provider adapters, and provider retry. |

Installing the package into pi (`pi install npm:@onkernel/loop`) registers the
Expand Down Expand Up @@ -373,11 +373,11 @@ themselves:
```ts
const catalog = compileLoopToolCatalog({
model: "anthropic:claude-opus-5",
requestedTools: tools, // Loop specs and plain pi-ai Tool declarations
requestedTools: tools, // Loop specs and plain declarations ({ name, description, parameters })
});

catalog.entries; // identities, fingerprints, declarations, coordinates
catalog.toolDeclarations; // pi-ai Tool declarations for Context.tools
catalog.toolDeclarations; // LoopToolDeclarations, structurally pi-ai Tools, for Context.tools
catalog.headers.merge(callerHeaders);
await catalog.payload.apply(payload, catalog.model);
catalog.incoming;
Expand Down
3 changes: 2 additions & 1 deletion packages/loop/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@
"@earendil-works/pi-ai": "0.83.0",
"@onkernel/sdk": "0.49.0",
"openai": "^6.26.0",
"sharp": "^0.35.3"
"sharp": "^0.35.3",
"typebox": "1.3.7"
},
"peerDependencies": {
"@earendil-works/pi-coding-agent": "*"
Expand Down
2 changes: 1 addition & 1 deletion packages/loop/src/core/actions/browser.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Type, type TSchema } from "@earendil-works/pi-ai";
import { Type, type TSchema } from "typebox";

/**
* Browser-plane canonical actions.
Expand Down
2 changes: 1 addition & 1 deletion packages/loop/src/core/actions/computer.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Type, type TSchema } from "@earendil-works/pi-ai";
import { Type, type TSchema } from "typebox";

/**
* Computer-plane canonical actions.
Expand Down
2 changes: 1 addition & 1 deletion packages/loop/src/core/actions/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TSchema } from "@earendil-works/pi-ai";
import type { TSchema } from "typebox";
import { BROWSER_ACTION_TYPES, createBrowserActionSchemaByType, type BrowserAction, type BrowserActionType, type BrowserActionSchemaOptions } from "./browser";
import { COMPUTER_ACTION_SCHEMA_BY_TYPE, COMPUTER_ACTION_TYPES, type ComputerAction, type ComputerActionType } from "./computer";

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { ComputerUseAction, MouseButton } from "../../../core/actions/index";
import type { ComputerUseAction, MouseButton } from "./actions/index";

interface NativeInput {
action: string;
Expand Down
13 changes: 6 additions & 7 deletions packages/loop/src/core/menu.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import type { Api, Model } from "@earendil-works/pi-ai";
import type { LoopCatalogModel, LoopModelFacts } from "./model-info";
import { loop } from "./tools";
import { getLoopModel, type LoopModelRef } from "../pi/models";
import { compileLoopToolCatalog, type LoopToolSpec } from "./tool-catalog";

/** Where a menu entry comes from, for grouping in a picker. */
Expand Down Expand Up @@ -40,17 +39,17 @@ export interface LoopToolMenuEntry {
* per-tool verdict.
*/
export function loopToolMenu(
model: LoopModelRef | Model<Api>,
model: LoopCatalogModel,
selected: readonly LoopToolSpec[] = [],
facts?: LoopModelFacts,
): LoopToolMenuEntry[] {
const resolved = typeof model === "string" ? getLoopModel(model) : model;
const selectedIdentities = new Set(selected.map((tool) => tool.identity));
return offerableEntries().map((entry) => {
const isSelected = entry.tools.every((tool) => selectedIdentities.has(tool.identity));
const candidate = isSelected
? [...selected]
: [...selected.filter((tool) => !entry.tools.some((offered) => offered.identity === tool.identity)), ...entry.tools];
const failure = compileFailure(resolved, candidate);
const failure = compileFailure(model, facts, candidate);
return {
key: entry.key,
label: entry.label,
Expand All @@ -64,9 +63,9 @@ export function loopToolMenu(
});
}

function compileFailure(model: Model<Api>, requestedTools: readonly LoopToolSpec[]): string | undefined {
function compileFailure(model: LoopCatalogModel, facts: LoopModelFacts | undefined, requestedTools: readonly LoopToolSpec[]): string | undefined {
try {
compileLoopToolCatalog({ model, requestedTools });
compileLoopToolCatalog({ model, requestedTools, facts });
return undefined;
} catch (error) {
return error instanceof Error ? error.message : String(error);
Expand Down
53 changes: 53 additions & 0 deletions packages/loop/src/core/model-info.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/** The model identity fields Loop's core consults. */
export interface LoopModelIdentity {
readonly provider: string;
readonly id: string;
}

/** A provider-native tool surface Loop can offer for a model. */
export type ComputerUseNativeSurface = "computer" | "browser";

/** Loop tool-catalog capabilities for a concrete model. */
export interface LoopModelCapabilities {
readonly acceptsComplexSchemas: boolean;
readonly acceptsLargeSchemas: boolean;
readonly serializesStateMutations: boolean;
}

/**
* Framework-neutral view of the model a catalog is compiled for: identity and
* the transport it carries. A pi-ai `Model` satisfies this shape structurally;
* core never sees more of it.
*/
export interface LoopCatalogModel extends LoopModelIdentity {
readonly api: string;
}

/**
* Per-model availability facts the compiler and menu consult. The binding
* supplies them — pi derives them from its model registry and quirk tables —
* so core never owns a provider capability lookup. Absent facts mean
* permissive capabilities and no native surfaces.
*/
export interface LoopModelFacts {
/** Request-shape limits for the model. Absent means permissive. */
readonly capabilities?: LoopModelCapabilities;
/** Provider-native tool surfaces the model can carry. Absent means none. */
readonly nativeSurfaces?: readonly ComputerUseNativeSurface[];
}

const PERMISSIVE_CAPABILITIES: LoopModelCapabilities = Object.freeze({
acceptsComplexSchemas: true,
acceptsLargeSchemas: true,
serializesStateMutations: false,
});

/** The capabilities a facts object carries, defaulting to permissive. */
export function loopModelFactsCapabilities(facts: LoopModelFacts | undefined): LoopModelCapabilities {
return facts?.capabilities ?? PERMISSIVE_CAPABILITIES;
}

/** The native surfaces a facts object carries, defaulting to none. */
export function loopModelFactsNativeSurfaces(facts: LoopModelFacts | undefined): readonly ComputerUseNativeSurface[] {
return facts?.nativeSurfaces ?? [];
}
46 changes: 30 additions & 16 deletions packages/loop/src/core/resources.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import type { AgentTool, AgentToolResult } from "@earendil-works/pi-agent-core";
import type { ImageContent, TextContent } from "@earendil-works/pi-ai";
import type Kernel from "@onkernel/sdk";
import type { ComputerUseAction } from "./actions/index";
import type { LoopCoordinateContract, LoopToolSpec } from "./tool-catalog";
Expand All @@ -22,7 +20,28 @@ export interface LoopExecutionDetails {
error?: string;
}

type ToolContent = Array<TextContent | ImageContent>;
/** One content block returned to the model by a materialized Loop tool. */
export type LoopToolResultContent =
| { type: "text"; text: string }
| { type: "image"; data: string; mimeType: string };

/** Framework-neutral result of executing a materialized Loop tool. */
export interface LoopToolExecutionResult {
content: LoopToolResultContent[];
details: LoopExecutionDetails;
}

/**
* Framework-neutral executable: a spec bound to this pool's browser. A
* framework binding wraps `execute` in its own tool shape; the executable
* itself only ever sees the model-provided input and an abort signal.
*/
export interface LoopExecutableTool {
readonly spec: LoopToolSpec;
execute(input: unknown, signal?: AbortSignal): Promise<LoopToolExecutionResult>;
}

type ToolContent = LoopToolResultContent[];

/**
* One per-agent browser resource pool. Tool catalogs may be rebuilt without
Expand All @@ -33,7 +52,7 @@ export class LoopExecutionResources {
readonly client: Kernel;
private readonly translator: InternalComputerTranslator;
/** Each spec is materialized exactly once per resource pool. */
private readonly materialized = new WeakMap<LoopToolSpec, AgentTool>();
private readonly materialized = new WeakMap<LoopToolSpec, LoopExecutableTool>();

constructor(options: {
browser: KernelBrowser;
Expand All @@ -46,17 +65,12 @@ export class LoopExecutionResources {
this.translator = new InternalComputerTranslator(options);
}

materialize(spec: LoopToolSpec): AgentTool {
materialize(spec: LoopToolSpec): LoopExecutableTool {
const cached = this.materialized.get(spec);
if (cached) return cached;
const definition = spec.declaration;
const tool: AgentTool = {
name: spec.name,
label: spec.name,
description: definition.description,
parameters: definition.parameters,
executionMode: "sequential",
execute: async (_toolCallId, input, signal) => {
const tool: LoopExecutableTool = {
spec,
execute: async (input, signal) => {
if (spec.execution.kind === "playwright") return this.executePlaywright(spec.name, input);
const actions = spec.execution.toActions(input);
return this.executeActions(spec, actions, signal);
Expand All @@ -82,7 +96,7 @@ export class LoopExecutionResources {
this.translator.dispose();
}

private async executeActions(spec: LoopToolSpec, actions: ComputerUseAction[], signal?: AbortSignal): Promise<AgentToolResult<LoopExecutionDetails>> {
private async executeActions(spec: LoopToolSpec, actions: ComputerUseAction[], signal?: AbortSignal): Promise<LoopToolExecutionResult> {
if (spec.execution.kind !== "actions") throw new Error(`tool "${spec.name}" has no action executor`);
let result: BatchExecutionResult;
let failure: BatchExecutionError | undefined;
Expand Down Expand Up @@ -124,7 +138,7 @@ export class LoopExecutionResources {
};
}

private async executePlaywright(name: string, input: unknown): Promise<AgentToolResult<LoopExecutionDetails>> {
private async executePlaywright(name: string, input: unknown): Promise<LoopToolExecutionResult> {
const parameters = asRecord(input);
const code = parameters.code;
if (typeof code !== "string") throw new Error(`${name} requires string code`);
Expand Down Expand Up @@ -207,7 +221,7 @@ function formatBrowserWaitResult(result: BrowserWaitForResult): string {
return [`wait_for: ${result.status}/${result.evidence}${reason} after ${result.elapsed_ms}ms`, ...result.details].join("\n");
}

function toImage(screenshot: { data: Buffer; mimeType: string }): ImageContent {
function toImage(screenshot: { data: Buffer; mimeType: string }): LoopToolResultContent {
return { type: "image", data: screenshot.data.toString("base64"), mimeType: screenshot.mimeType };
}

Expand Down
Loading
Loading