diff --git a/.claude/skills/plotjuggler-plugin/SKILL.md b/.claude/skills/plotjuggler-plugin/SKILL.md index 61ac33a3..11b57afb 100644 --- a/.claude/skills/plotjuggler-plugin/SKILL.md +++ b/.claude/skills/plotjuggler-plugin/SKILL.md @@ -2,16 +2,14 @@ name: plotjuggler-plugin description: >- Write or modify a PlotJuggler plugin (DataSource, MessageParser, Toolbox, or - Dialog) using the plotjuggler_sdk C++20 SDK. Use this whenever the task is to - implement, extend, debug, or build a PlotJuggler plugin — importing a file - format, streaming live data, parsing message payloads, adding a data-processing - toolbox, or building a plugin configuration dialog — even if the user does not - say the word "plugin" (e.g. "add a CSV loader to PlotJuggler", "make PJ read my - MCAP", "parse my protobuf topic", "a dialog for my source"). It gives the fast - path (which base class, which header, which macro), the correct CMake for both - in-tree and installed-SDK builds, and the load-bearing ABI/lifetime rules that - are easy to get wrong. Do NOT use it for changing the SDK's own ABI/protocol - (that is a maintainer task with different rules — see the note below). + Dialog) or native functional parser module using the plotjuggler_sdk. Use this + whenever the task is to implement, extend, debug, or build a PlotJuggler + extension — importing a file format, streaming live data, parsing message + payloads, mapping a custom type to a canonical object or scalars, adding a + data-processing toolbox, or building a plugin configuration dialog — even if + the user does not say the word "plugin". It gives the fast path, correct CMake, + and load-bearing ABI/lifetime rules. Do NOT use it for changing the SDK's own + ABI/protocol (that is a maintainer task with different rules — see below). --- # Writing a PlotJuggler plugin @@ -23,6 +21,12 @@ the C-ABI plumbing (entry point, version symbol, exception trampolines, symbol folding) so **you never touch the raw ABI**. Your job is to override a handful of virtual methods and ship the library. +There is also a lighter extension shape: a **functional parser module** is one +C++17 source compiled as a native shared library, with no plugin-family +boilerplate. Write a MessageParser plugin when you own a whole encoding. Write a +parser module when you only need one or a few custom message types rendered as +canonical objects or scalars. Start at `references/parser-module.md`. + This skill is the author-oriented fast path. The repo's `pj_plugins/docs/` guides are the full reference; this steers you to the right one and front-loads the things that silently break a plugin. @@ -66,6 +70,7 @@ plotjuggler_sdk::plugin_sdk)` — only the acquisition step differs: |---|---|---| | Turn a **file** or a **live source** (socket, serial, hardware) into topics | **DataSource** | `references/data-source.md` | | Decode a **byte payload** on a topic into named fields (JSON, protobuf, ROS, custom) | **MessageParser** | `references/message-parser.md` | +| Map **one or a few message types** to canonical objects or scalars without owning the encoding | **Functional parser module** | `references/parser-module.md` | | A **tool** that reads existing data, transforms it, and writes new topics | **Toolbox** | `references/toolbox.md` | | A **configuration UI** (for a source/parser/toolbox, or standalone) | **Dialog** | `references/dialog.md` | @@ -95,8 +100,8 @@ second argument is the **manifest JSON literal** (see Step 3). > ⚠ **Header-location trap.** Three base classes live under `pj_base/sdk/`, but > `MessageParserPluginBase` lives under **`pj_plugins/sdk/`**, and the Dialog SDK > lives under `pj_plugins/sdk/` too (installed there from the `dialog_protocol` -> module). Some in-repo docs still show the parser header under `pj_base/` — that -> is stale; use the paths in the table. +> module). Do not substitute a `pj_base/sdk/` path for either one; use the paths +> in the table. The quickest correct start is to copy a working example from this repo and edit it: `examples/sdk_consumer/` (minimal external DataSource with the full CMake), @@ -223,6 +228,7 @@ coalescing, etc.) — read them. | What each family may do; capabilities; permission matrix; config contract | `pj_plugins/docs/REQUIREMENTS.md` | | How the C ABI / loaders / host bridge work (mostly maintainer detail) | `pj_plugins/docs/ARCHITECTURE.md` | | Writing each family, in depth | `pj_plugins/docs/{data-source,message-parser,toolbox,dialog-plugin}-guide.md` | +| Authoring a native functional parser module | `references/parser-module.md`, `pj_base/include/pj_base/parser_module/README.md` | | Dialog `WidgetData` setters + event-handler signatures | `docs/dialog-sdk-reference.md` | | Builtin object types + their wire codecs | `docs/builtin_type.md`, `pj_base/include/pj_base/builtin/` | | Object store: publish/read objects, ownership, lazy fetch | `V4_STORE.md` | diff --git a/.claude/skills/plotjuggler-plugin/references/builtin-objects.md b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md index 814e2df9..a1cdc6be 100644 --- a/.claude/skills/plotjuggler-plugin/references/builtin-objects.md +++ b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md @@ -66,8 +66,8 @@ if (!st) return PJ::unexpected(st.error()); Each builtin has a `*_codec.hpp` under `PJ::` (not `PJ::sdk::`) with `serializeXxx()` / `deserializeXxx()`. Note the asymmetry: `serializeXxx()` returns a plain `std::vector` (serialization can't fail), while `deserializeXxx()` -returns `PJ::Expected<...>`. The `RobotDescription` type is the exception — it -carries its source text as-is and has no codec. +returns `PJ::Expected<...>`. `RobotDescription` also has a canonical codec; its +source text remains unchanged inside the small wire envelope. **MessageParsers emit objects differently — by returning, not pushing.** A parser's `SchemaHandler.parse_object` returns an `ObjectRecord{optional ts, diff --git a/.claude/skills/plotjuggler-plugin/references/message-parser.md b/.claude/skills/plotjuggler-plugin/references/message-parser.md index 9084ce4f..401702f3 100644 --- a/.claude/skills/plotjuggler-plugin/references/message-parser.md +++ b/.claude/skills/plotjuggler-plugin/references/message-parser.md @@ -14,6 +14,10 @@ not a parser. A parser only decodes payloads the host hands it. A parser is righ when many payloads on a topic share an encoding (JSON, protobuf, ROS, a custom binary) and you decode each into fields. +If an existing encoding already carries one custom type that you want to render +as a canonical object or scalars, use a **functional parser module** instead of +forking or replacing the encoding's MessageParser. See `parser-module.md`. + ## Header (trap) ```cpp @@ -21,8 +25,7 @@ binary) and you decode each into fields. ``` This base class lives under `pj_plugins/sdk/`, unlike the DataSource/Toolbox bases -which live under `pj_base/sdk/`. Some in-repo docs show it under `pj_base/` — that -is stale. +which live under `pj_base/sdk/`. Do not substitute a `pj_base/sdk/` include. ## The current model: register SchemaHandlers, don't override parse() @@ -96,6 +99,24 @@ Key pieces: mapping type names → handlers scales well (the official ROS parser maps 20+ types this way). +## Route claims and functional extensions + +Recompile a handler-registering parser against SDK 0.22 and the base class +automatically exposes all three host-facing tables: + +- `pj.parser_route_claims.v1` reports exact claims from the handler table. It + never reports a wildcard. The host synthesizes one universal wildcard scalar + claim per manifest encoding. +- `pj.parser_functional.v1` carries complete canonical-wire objects and scalar + records through synchronous caller-owned sinks. +- `pj.parser_functional.v2` keeps the scalar route unchanged and adds the object + splice sink. The host asks for v2 first and falls back to v1. + +Scalar and object routes resolve independently. Another provider may therefore +own the object route for a type that this parser still flattens to scalars. There +is no extra override or registration API: populate the `SchemaHandler` table and +the base class derives these extensions. + ## Optional overrides - `loadConfig(json)` / `saveConfig()` — parser options (array-size limits, diff --git a/.claude/skills/plotjuggler-plugin/references/parser-module.md b/.claude/skills/plotjuggler-plugin/references/parser-module.md new file mode 100644 index 00000000..3c8ab32b --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/parser-module.md @@ -0,0 +1,193 @@ +# Functional parser module + +A functional parser module maps one or a few message types from an existing +encoding to canonical PlotJuggler objects or scalar fields. It is a single C++17 +source compiled as a native shared library, with no plugin-family vtable or +MessageParser boilerplate. + +Use a **MessageParser plugin** when you own an encoding and must decode its broad +type universe. Use a **parser module** when the encoding already exists and you +only need an exact custom type rendered as an object or scalars. + +## Minimal native module + +```cpp +#include + +#include +#include + +class RawMonoImageParser final : public pj::FunctionalParser { + public: + pj::Status bind(const pj::BindingInfo& info) override { + if (info.route() != pj::Route::kObject || + info.expectedObjectType() != pj::ObjectWriter::kImageObjectType) { + return pj::Status::decline("this module only produces Image objects"); + } + return pj::Status::ok(); + } + + pj::Status parseObject(pj::PayloadView payload, pj::Timestamp timestamp, + pj::ObjectWriter& output) override { + if (payload.size > std::numeric_limits::max()) { + return pj::Status::error("image row is too large"); + } + + auto image = output.image(); + if (timestamp.has_value) { + if (auto status = image.setTimestamp(timestamp.nanoseconds); + !status.isOk()) { + return status; + } + } + const auto width = static_cast(payload.size); + if (auto status = image.setWidth(width); !status.isOk()) { + return status; + } + if (auto status = image.setHeight(1); !status.isOk()) { + return status; + } + if (auto status = image.setEncoding("mono8"); !status.isOk()) { + return status; + } + if (auto status = image.setRowStep(width); !status.isOk()) { + return status; + } + return image.setData(payload); + } +}; + +PJ_FUNCTIONAL_PARSER(RawMonoImageParser) +``` + +`Timestamp` is per-message input. Check `has_value` before using +`nanoseconds`. Override `parseScalars(PayloadView, Timestamp, ScalarWriter&)` +instead, or as well, when the manifest claims the scalar route. + +## Claims manifest + +```json +{ + "module_abi": 1, + "id": "com.example.raw-mono-image", + "name": "Raw mono image parser", + "version": "1.0.0", + "claims": [ + { + "claim_id": "raw-image-v1", + "encoding": "ros2msg", + "type_name": "example_msgs/msg/RawImage", + "routes": ["object"], + "object_type": "kImage", + "schema_digests": [ + "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + ], + "priority": 0 + } + ] +} +``` + +- `module_abi` must equal the frozen parser-module ABI version (`1`). +- `id` is the stable provider identity. Never change it for a display rename. +- `name` is display-only metadata. +- `version` is a valid SemVer string. +- `claim_id` is stable and unique within `id`; together they form claim identity. +- `encoding` is a case-sensitive SDK-registered encoding such as `ros2msg` or + `protobuf`. +- `type_name` is encoding-normalized: `pkg/msg/Type` for `ros2msg`, or the full + dotted protobuf message name. Modules claim exact types, not `"*"` objects. +- `routes` is a non-empty array containing `"scalar"`, `"object"`, or both. +- `object_type` is required with the object route and forbidden without it. Use + the exact `BuiltinObjectType` spelling, such as `kImage`. +- `schema_digests` is optional. Each entry is `sha256:` plus 64 hexadecimal + digits; an empty or omitted set accepts any schema digest at catalog matching. +- `priority` is required and must be in `[-1000, 1000]`. Host-owned provenance + tiers outrank priority, so do not use priority as a trust signal. + +Manifest order fixes each claim's `claimIndex()`. The native loader copies the +embedded bytes; catalog ingestion validates the complete JSON transactionally. + +## Build + +```cmake +find_package(plotjuggler_sdk 0.22 REQUIRED COMPONENTS parser_module) + +pj_add_parser_module(raw_mono_image_parser + SOURCE raw_mono_image_parser.cpp + MANIFEST raw_mono_image_parser.module.json + TARGETS native +) +``` + +The helper embeds the manifest, hides every non-ABI symbol, and exports the +complete native `pj_module_*` set. SDK 0.22 accepts only `TARGETS native`; +requesting `TARGETS wasm` stops configuration with “wasm support arrives with +the SDK wasm loader milestone”. The shipped WASI check is structural +conformance testing, not a wasm authoring or execution target. + +## Choose a schema-compatibility strategy + +Use the least brittle rung that fits the format: + +1. **Hardcode one layout** only when the wire schema is immutable and externally + versioned. +2. **Digest allow-list.** Put accepted `sha256:<64-hex>` values in the claim and + return `Status::decline(...)` from `bind()` for an unsupported revision, so + the resolver may try another claim. +3. **Inspect at bind.** Compile requested paths once with `CdrFieldLocator` from + a ROS 2 concatenated `.msg` bundle, or `ProtoFieldLocator` from a serialized + `FileDescriptorSet`, then reuse the plan for every message. + +`CdrFieldLocator` traverses fixed arrays, bounded/unbounded sequences, bounded +strings, and string arrays/sequences. Cyclic schemas and traversal deeper than +64 levels fail at bind with an error `Status`. The CDR and protobuf readers also +bounds-check every per-message traversal. + +## ObjectWriter builders and splices + +The nine builders with frozen splice-eligible bulk fields are: + +- `image()` +- `pointCloud()` +- `depthImage()` +- `occupancyGrid()` +- `compressedPointCloud()` +- `mesh3D()` +- `videoFrame()` +- `occupancyGridUpdate()` +- `voxelGrid()` + +Every builder has `setData(PayloadView)`, which copies the bytes into the full +canonical wire object. Every builder also has +`setDataFromInput(InputSpanRef)`, which omits that one bulk field from the wire +and records one splice into the exact parse payload. Obtain a safe reference +from `CdrReader::spanRef(...)` when possible. Splice offsets are relative to the +payload start, not the CDR encapsulation or a nested field; the writer rejects +out-of-range, repeated, or mixed copy/splice selection before it can emit an +invalid descriptor. + +## Traps + +- The kit is header-only and WASI-clean: no threads, filesystem, iostream, host + SDK linkage, or exceptions across its API. In SDK 0.22 the supported build + product is nevertheless native-only. +- Return `pj::Status` / `pj::Expected`; do not throw. `Blob` uses nothrow + allocation and protobuf matching is bounded, so allocation failure is a + reported data error rather than a process abort or contract strike. +- Every view in `BindingInfo` is borrowed until `bind()` returns. Call + `info.owningCopy()` and retain the returned `OwnedBindingInfo` if later parses + need any field. +- `Status::decline(...)` from bind means “this valid claim does not accept this + binding”. A module-reported negative parse result, including malformed + application data or allocation failure, is a data error and does not earn a + strike. +- Bad/stale-token returns, a successful call that supplies a malformed output + descriptor, wrong route or object type, or an ineligible/out-of-bounds splice + are contract violations. Native hosts feed those results to + `ParserModuleStrikeTracker`: three strikes quarantine the claim; after one + successful create/bind replay, another three-strike cycle disables it for the + session. The tracker is host-driven state, not module-author API. +- Instance tokens are generated index+generation values. Token `0` is reserved + for creation errors; stale tokens are rejected and leave a retrievable + diagnostic. Do not derive meaning from token bits. diff --git a/.github/workflows/linux-ci.yml b/.github/workflows/linux-ci.yml index 2b8da885..cbf1f071 100644 --- a/.github/workflows/linux-ci.yml +++ b/.github/workflows/linux-ci.yml @@ -146,6 +146,21 @@ jobs: # abigail-tools provides abidw/abidiff for the ABI drift gate (below). run: sudo apt-get update && sudo apt-get install -y xvfb ccache ninja-build zstd abigail-tools + - name: Install wasi-sdk 27 + shell: bash + run: | + set -euo pipefail + version="27.0" + archive="${RUNNER_TEMP}/wasi-sdk-${version}.tar.gz" + install_dir="${RUNNER_TEMP}/wasi-sdk-${version}" + url="https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-27/wasi-sdk-${version}-x86_64-linux.tar.gz" + + curl -fsSL --retry 5 --retry-all-errors "${url}" -o "${archive}" + mkdir -p "${install_dir}" + tar -xzf "${archive}" --strip-components=1 -C "${install_dir}" + echo "PJ_WASI_SDK_ROOT=${install_dir}" >> "${GITHUB_ENV}" + "${install_dir}/bin/clang++" --version + - name: Configure ccache # Compiler-output cache for our own C++ — complementary to the Conan # cache (which holds prebuilt third-party packages, not our objects). @@ -207,11 +222,17 @@ jobs: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache + -DPJ_WASI_SDK_ROOT=${PJ_WASI_SDK_ROOT} -DPJ_ENABLE_ABI_CHECK=ON - name: Build run: cmake --build build + - name: WASI parser-module conformance + run: | + cmake --build build --target parser_module_wasm_conformance_fixture + ctest --test-dir build -L wasi --output-on-failure + - name: ABI drift gate # Mechanically enforces the Release Versioning policy (CLAUDE.md): a non-MAJOR # change must not break ABI. Diffs the mock_data_source_plugin canary DSO against diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bd9815e..ee15478d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,46 @@ All notable changes to `plotjuggler_sdk` are recorded here. Versioning policy is in [`CLAUDE.md`](./CLAUDE.md) → "Release Versioning". +## [0.22.0] + +### Feature: extensible parser routing and functional parser modules (MINOR) + +Parser selection can now be described, resolved, and executed through stable, +additive SDK contracts: + +- The new `pj.parser_route_claims.v1` extension reports exact scalar/object + handler claims, while `pj.parser_functional.v2` adds the object splice sink + without changing the frozen v1 declarations. `MessageParserHandle` negotiates + v2 first, falls back to v1, reconstructs splices, and rejects objects whose + type differs from the selected claim. +- The parser-module ABI defines lifecycle exports, little-endian binding/input/ + output codecs, canonical-object splice eligibility, native manifest metadata + exports, and the frozen wasm manifest custom-section name and byte codec. +- The host claim catalog validates module manifests and synthesized plugin + claims. Its deterministic resolver applies exact/wildcard, provenance, + priority, pin, probe-cache, and fail-closed selection policy. +- The native loader and per-instance runtime validate exports, lifecycle + results, descriptors, and splices, reconstruct spliced canonical objects, and + expose contract-strike quarantine and session-disable state. Module tokens + are synchronized index+generation values, so stale tokens fail with a + diagnostic rather than aliasing a new instance. +- The standalone header-only C++17 authoring kit provides bounded CDR/protobuf + readers and field locators, time normalization, and the complete native + functional-module export wrapper. Parse callbacks receive the per-message + `Timestamp`; CDR plans support bounded sequences/strings and string + arrays/sequences while rejecting cyclic or over-depth schemas at bind. + `ObjectWriter` covers all nine splice-eligible canonical types: Image, + PointCloud, DepthImage, OccupancyGrid, CompressedPointCloud, Mesh3D, + VideoFrame, OccupancyGridUpdate, and VoxelGrid. Bulk allocations are fallible + under `-fno-exceptions` and report data errors. +- A shared wasm custom-section codec embeds exact manifest bytes. The wasi-sdk + 27 compile gate statically audits reactor exports and their frozen wasm + signatures, manifest delivery, and absence of native-only metadata exports; + wasm loading, execution, and `pj_add_parser_module(... TARGETS wasm)` are not + part of this release. + +All additions preserve the existing plugin ABI and protocol versions. + ## [0.21.0] ### Fix: convenience registerService honors its documented assertion (PATCH) diff --git a/CLAUDE.md b/CLAUDE.md index 7ce1957a..61cb3c6a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,8 +4,8 @@ PlotJuggler SDK — C++20 foundation libraries that make up the PlotJuggler plugin SDK and host-side plugin loading. **Read-only submodule** inside PJ4: consumed as-is; changes happen in this repo, -not in the PJ4 superproject. This file is the single navigation node for the whole submodule — the -two modules below have no own CLAUDE.md. +not in the PJ4 superproject. This file is the root navigation node for the whole submodule; +`pj_plugins/CLAUDE.md` adds module-specific guidance, while `pj_base` has no separate CLAUDE.md. > The columnar storage engine (`pj_datastore`) used to live here. It now lives in the PlotJuggler > application repo as a top-level module: plugins reach storage only through the C ABI defined in @@ -15,13 +15,17 @@ two modules below have no own CLAUDE.md. ### Modules - **pj_base** — vocabulary types (`Timestamp`, `DatasetId`, `Expected`, `Span`, type trees), - the canonical builtin object vocabulary (`pj_base/builtin/`: 16 struct headers — Image, DepthImage, + the canonical builtin object vocabulary (`pj_base/builtin/`: 17 struct headers — Image, DepthImage, PointCloud, CompressedPointCloud, OccupancyGrid(+Update), Mesh3D, VideoFrame, - SceneEntities, RobotDescription, CameraInfo, Log, ImageAnnotations, FrameTransforms, PosesInFrame, VoxelGrid) and their 15 - wire codecs (RobotDescription carries source text as-is — no codec), the C-ABI protocol headers for - DataSource/MessageParser/Toolbox + the C++ SDK base classes / host-view helpers built on them. + SceneEntities, RobotDescription, CameraInfo, Log, ImageAnnotations, FrameTransforms, PosesInFrame, + VoxelGrid, PlotMarkers) and their canonical wire codecs, the C-ABI protocol headers for + DataSource/MessageParser/Toolbox + the C++ SDK base classes / host-view helpers built on them, the + standalone C++17 functional parser-module authoring kit (`pj_base/parser_module/`), the host-side + wasm parser-module manifest custom-section codec, and the test-only static WASI ABI auditor. The + 0.22 authoring helper builds native parser modules only; wasm loading/execution is not present. - **pj_plugins** — host-side loaders + RAII handles + plugin **discovery** (directory scan + embedded-manifest inspection) for four plugin families (DataSource, MessageParser, Dialog, Toolbox), + parser claim admission/resolution and native functional parser-module execution, config-envelope helpers, and the **dialog C ABI** (`pj_plugins/dialog_protocol/`). The duplicate-resolution *catalog* (which copy wins by priority/version/compatibility) is host policy and lives in the app (`pj_runtime`), built on these discovery primitives. Note the split: the DataSource/MessageParser/Toolbox C-ABI @@ -60,6 +64,8 @@ documentation check before commit. **Plugin system** (`pj_plugins/docs/`): `REQUIREMENTS.md` (families, capability system, config contract) · `ARCHITECTURE.md` (C ABI protocols, SDK base classes, host loaders, dialog protocol) · `data-source-guide.md` · `message-parser-guide.md` · `dialog-plugin-guide.md` · `toolbox-guide.md`. +The concise native functional-module authoring reference is +`.claude/skills/plotjuggler-plugin/references/parser-module.md`. ## Build & Test diff --git a/CMakeLists.txt b/CMakeLists.txt index a91ec1f2..8cae0b95 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,6 +1,11 @@ cmake_minimum_required(VERSION 3.22) file(READ "${CMAKE_CURRENT_LIST_DIR}/VERSION" PJ_SDK_VERSION) string(STRIP "${PJ_SDK_VERSION}" PJ_SDK_VERSION) +# Editing VERSION must re-run configure; otherwise an existing build tree keeps +# stamping the previous version into the generated pj_base/sdk/version.hpp, and +# min_sdk_required gating then compares against a version the SDK never shipped. +set_property( + DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${CMAKE_CURRENT_LIST_DIR}/VERSION") project(plotjuggler_sdk VERSION "${PJ_SDK_VERSION}" LANGUAGES CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) @@ -10,6 +15,7 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON) list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}/cmake") include(GNUInstallDirs) # CMAKE_INSTALL_LIBDIR, etc. used by PjPluginManifest include(PjPluginManifest) +include(PjParserModule) # --------------------------------------------------------------------------- # Options @@ -122,6 +128,7 @@ endif() # Exported CMake namespace: plotjuggler_sdk:: # Components: # base — vocabulary types (always available) +# parser_module — standalone header-only functional module authoring kit # plugin_sdk — plugin-author surface: base + dialog SDK + parser SDK # plugin_host — host-side loaders (data_source, message_parser, toolbox, # dialog, catalogs) @@ -153,6 +160,7 @@ if(PJ_INSTALL_SDK) "${CMAKE_CURRENT_BINARY_DIR}/plotjuggler_sdkConfig.cmake" "${CMAKE_CURRENT_BINARY_DIR}/plotjuggler_sdkConfigVersion.cmake" cmake/PjPluginManifest.cmake + cmake/PjParserModule.cmake DESTINATION ${PJ_PACKAGE_CMAKE_DIR} ) endif() diff --git a/VERSION b/VERSION index 88541566..21574090 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.21.0 +0.22.0 diff --git a/cmake/PjParserModule.cmake b/cmake/PjParserModule.cmake new file mode 100644 index 00000000..b70d6847 --- /dev/null +++ b/cmake/PjParserModule.cmake @@ -0,0 +1,102 @@ +# PjParserModule.cmake +# +# Native functional parser-module target helper. Wasm manifest embedding and +# toolchain support arrive with the wasm loader milestone. + +function(pj_add_parser_module TARGET) + set(_options) + set(_oneValueArgs SOURCE MANIFEST) + set(_multiValueArgs TARGETS) + cmake_parse_arguments(ARG "${_options}" "${_oneValueArgs}" "${_multiValueArgs}" ${ARGN}) + + if(ARG_UNPARSED_ARGUMENTS) + message(FATAL_ERROR + "pj_add_parser_module(${TARGET}): unknown arguments: ${ARG_UNPARSED_ARGUMENTS}") + endif() + if(NOT ARG_SOURCE) + message(FATAL_ERROR "pj_add_parser_module(${TARGET}): SOURCE is required") + endif() + if(NOT ARG_MANIFEST) + message(FATAL_ERROR "pj_add_parser_module(${TARGET}): MANIFEST is required") + endif() + if(NOT ARG_TARGETS) + message(FATAL_ERROR "pj_add_parser_module(${TARGET}): TARGETS native is required") + endif() + foreach(_requested_target IN LISTS ARG_TARGETS) + if(NOT _requested_target STREQUAL "native") + message(FATAL_ERROR + "pj_add_parser_module(${TARGET}): TARGETS ${_requested_target} is unavailable; " + "wasm support arrives with the SDK wasm loader milestone") + endif() + endforeach() + + get_filename_component(_module_source "${ARG_SOURCE}" ABSOLUTE BASE_DIR "${CMAKE_CURRENT_SOURCE_DIR}") + get_filename_component(_module_manifest "${ARG_MANIFEST}" ABSOLUTE BASE_DIR "${CMAKE_CURRENT_SOURCE_DIR}") + if(NOT EXISTS "${_module_source}") + message(FATAL_ERROR "pj_add_parser_module(${TARGET}): SOURCE not found: ${_module_source}") + endif() + if(NOT EXISTS "${_module_manifest}") + message(FATAL_ERROR "pj_add_parser_module(${TARGET}): MANIFEST not found: ${_module_manifest}") + endif() + + set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${_module_manifest}") + file(READ "${_module_manifest}" _manifest_json) + string(JSON _claims_type ERROR_VARIABLE _claims_error TYPE "${_manifest_json}" claims) + if(_claims_error OR NOT _claims_type STREQUAL "ARRAY") + message(FATAL_ERROR + "pj_add_parser_module(${TARGET}): MANIFEST must contain a claims array") + endif() + string(JSON _claim_count LENGTH "${_manifest_json}" claims) + if(_manifest_json MATCHES "\\)PJM\"") + message(FATAL_ERROR + "pj_add_parser_module(${TARGET}): MANIFEST contains the reserved raw-string delimiter") + endif() + + set(_generated_dir "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_parser_module_generated") + file(MAKE_DIRECTORY "${_generated_dir}") + set(_manifest_header "${_generated_dir}/${TARGET}_manifest.hpp") + file(WRITE "${_manifest_header}" + "#pragma once\n" + "#define PJ_PARSER_MODULE_HAS_MANIFEST 1\n" + "namespace pj { namespace detail {\n" + "inline constexpr char kBuiltManifest[] = R\"PJM(${_manifest_json})PJM\";\n" + "} }\n" + "#define PJ_PARSER_MODULE_CLAIM_COUNT ${_claim_count}\n") + + add_library(${TARGET} MODULE "${_module_source}" "${_manifest_header}") + target_link_libraries(${TARGET} PRIVATE plotjuggler_sdk::parser_module) + target_include_directories(${TARGET} PRIVATE "${_generated_dir}") + target_compile_definitions(${TARGET} PRIVATE + PJ_PARSER_MODULE_MANIFEST_HEADER=\"${TARGET}_manifest.hpp\") + set_target_properties(${TARGET} PROPERTIES + CXX_STANDARD 17 + CXX_STANDARD_REQUIRED YES + CXX_EXTENSIONS NO + CXX_VISIBILITY_PRESET hidden + C_VISIBILITY_PRESET hidden + VISIBILITY_INLINES_HIDDEN YES + ) + if(DEFINED PJ_WARNING_FLAGS) + target_compile_options(${TARGET} PRIVATE ${PJ_WARNING_FLAGS}) + endif() + + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(_version_script "${_generated_dir}/${TARGET}.map") + file(WRITE "${_version_script}" + "{\n global:\n pj_module_abi;\n pj_module_create;\n pj_module_destroy;\n" + " pj_module_bind;\n pj_module_parse;\n pj_module_last_error;\n" + " pj_module_alloc;\n pj_module_free;\n pj_module_manifest_addr;\n" + " pj_module_manifest_len;\n local: *;\n};\n") + target_link_options(${TARGET} PRIVATE + "LINKER:-z,defs" + "LINKER:--exclude-libs,ALL" + "LINKER:--version-script=${_version_script}") + elseif(APPLE) + set(_exported_symbols "${_generated_dir}/${TARGET}.exports") + file(WRITE "${_exported_symbols}" + "_pj_module_abi\n_pj_module_create\n_pj_module_destroy\n_pj_module_bind\n" + "_pj_module_parse\n_pj_module_last_error\n_pj_module_alloc\n_pj_module_free\n" + "_pj_module_manifest_addr\n_pj_module_manifest_len\n") + target_link_options(${TARGET} PRIVATE "LINKER:-exported_symbols_list,${_exported_symbols}") + endif() +endfunction() diff --git a/cmake/plotjuggler_sdkConfig.cmake.in b/cmake/plotjuggler_sdkConfig.cmake.in index f6343655..0f62038e 100644 --- a/cmake/plotjuggler_sdkConfig.cmake.in +++ b/cmake/plotjuggler_sdkConfig.cmake.in @@ -16,6 +16,11 @@ foreach(_comp ${plotjuggler_sdk_FIND_COMPONENTS}) if(_comp STREQUAL "base") set(plotjuggler_sdk_base_FOUND TRUE) + elseif(_comp STREQUAL "parser_module") + # Header-only authoring surface plus its native module target helper. + include("${CMAKE_CURRENT_LIST_DIR}/PjParserModule.cmake") + set(plotjuggler_sdk_parser_module_FOUND TRUE) + elseif(_comp STREQUAL "plugin_sdk") find_dependency(nlohmann_json) # Ship the cmake/PjPluginManifest.cmake helper so plugin authors can call diff --git a/docs/builtin_type.md b/docs/builtin_type.md index 870da104..5915f68b 100644 --- a/docs/builtin_type.md +++ b/docs/builtin_type.md @@ -84,10 +84,12 @@ are small enough that the zero-copy anchor pattern is unnecessary. **Serialize only at an explicit boundary.** Large byte-backed types remain views over source-native payload bytes while they are inside one component and an ownership anchor is available. Every stable builtin also has a canonical -codec for storage/replay and for the `pj.parser_functional.v1` DSO boundary. +codec for storage/replay and for the `pj.parser_functional.v1`/v2 DSO boundary. That boundary deliberately serializes the value rather than sharing C++ class, -allocator, RTTI, or destructor state with the host. The schema and wire-format -details stay private; public SDK headers expose only SDK structs. +allocator, RTTI, or destructor state with the host. Functional v2 may carry one +frozen eligible bulk field as a splice into the input payload; the host still +reconstructs a canonical SDK object. The schema and wire-format details stay +private; public SDK headers expose only SDK structs. ## Serialization Families diff --git a/pj_base/CMakeLists.txt b/pj_base/CMakeLists.txt index 07589436..758d80b5 100644 --- a/pj_base/CMakeLists.txt +++ b/pj_base/CMakeLists.txt @@ -29,6 +29,8 @@ add_library(pj_base STATIC src/builtin/video_frame_codec.cpp src/builtin/voxel_grid_codec.cpp src/number_parse.cpp + src/parser_module_abi.cpp + src/parser_module_manifest.cpp src/semver.cpp src/type_tree.cpp src/data_source_host_views.cpp @@ -66,12 +68,24 @@ if(PJ_ASSERT_THROWS) target_compile_definitions(pj_base PUBLIC PJ_ASSERT_THROWS) endif() +# Standalone, header-only functional parser-module authoring kit. This target +# intentionally carries include paths and a C++ floor only: modules link no SDK +# library and remain suitable for native and WASI reactor builds. +add_library(pj_parser_module INTERFACE) +target_include_directories(pj_parser_module INTERFACE + $ + $ +) +target_compile_features(pj_parser_module INTERFACE cxx_std_17) +set_target_properties(pj_parser_module PROPERTIES EXPORT_NAME parser_module) +add_library(plotjuggler_sdk::parser_module ALIAS pj_parser_module) + # --------------------------------------------------------------------------- # Install (guarded by PJ_INSTALL_SDK in root CMakeLists.txt) # --------------------------------------------------------------------------- if(PJ_INSTALL_SDK) - install(TARGETS pj_base EXPORT plotjuggler_sdkTargets + install(TARGETS pj_base pj_parser_module EXPORT plotjuggler_sdkTargets ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} ) @@ -96,6 +110,8 @@ if(PJ_BUILD_TESTS) tests/number_parse_test.cpp tests/semver_test.cpp tests/plugin_data_api_test.cpp + tests/parser_module_abi_test.cpp + tests/parser_module_manifest_test.cpp tests/data_processors_api_test.cpp tests/settings_store_host_test.cpp tests/parser_runtime_host_test.cpp @@ -145,4 +161,108 @@ if(PJ_BUILD_TESTS) target_link_libraries(${test_name} PRIVATE pj_base GTest::gtest_main) add_test(NAME ${test_name} COMMAND ${test_name}) endforeach() + + # C++17 standalone authoring-kit gate. This target intentionally does not + # link pj_base; GTest is test infrastructure only. + add_executable(parser_module_kit_test tests/parser_module_kit_test.cpp) + set_target_properties(parser_module_kit_test PROPERTIES + CXX_STANDARD 17 + CXX_STANDARD_REQUIRED YES + CXX_EXTENSIONS NO + ) + target_compile_options(parser_module_kit_test PRIVATE ${PJ_WARNING_FLAGS}) + target_link_libraries(parser_module_kit_test PRIVATE pj_parser_module GTest::gtest_main) + add_test(NAME parser_module_kit_test COMMAND parser_module_kit_test) + + add_executable(parser_module_object_writer_test tests/parser_module_object_writer_test.cpp) + target_compile_definitions(parser_module_object_writer_test PRIVATE PJ_PARSER_MODULE_CLAIM_COUNT=1) + target_compile_options(parser_module_object_writer_test PRIVATE ${PJ_WARNING_FLAGS}) + target_link_libraries(parser_module_object_writer_test PRIVATE pj_parser_module pj_base GTest::gtest_main) + add_test(NAME parser_module_object_writer_test COMMAND parser_module_object_writer_test) + + # Static wasm ABI conformance. The authoring source is compiled directly by + # wasi-sdk, then a host-side auditor embeds and validates the final module. + set(PJ_WASI_SDK_ROOT "$ENV{PJ_WASI_SDK_ROOT}" CACHE PATH + "wasi-sdk 27 root used for parser-module conformance tests") + set(_pj_wasi_clang "${PJ_WASI_SDK_ROOT}/bin/clang++") + set(_pj_wasi_sysroot "${PJ_WASI_SDK_ROOT}/share/wasi-sysroot") + if(NOT PJ_WASI_SDK_ROOT) + message(STATUS + "Parser-module WASI conformance skipped: PJ_WASI_SDK_ROOT cache/env variable is not set") + elseif(NOT EXISTS "${_pj_wasi_clang}" OR NOT IS_DIRECTORY "${_pj_wasi_sysroot}") + message(STATUS + "Parser-module WASI conformance skipped: wasi-sdk is missing under PJ_WASI_SDK_ROOT=${PJ_WASI_SDK_ROOT}") + else() + file(STRINGS "${PJ_WASI_SDK_ROOT}/VERSION" _pj_wasi_version LIMIT_COUNT 1) + if(NOT _pj_wasi_version MATCHES "^27\\.") + message(FATAL_ERROR + "Parser-module WASI conformance requires wasi-sdk 27; found '${_pj_wasi_version}'") + endif() + message(STATUS + "Parser-module WASI conformance enabled with wasi-sdk ${_pj_wasi_version}: ${PJ_WASI_SDK_ROOT}") + + add_executable(parser_module_wasm_audit tests/parser_module_wasm_audit.cpp) + target_compile_options(parser_module_wasm_audit PRIVATE ${PJ_WARNING_FLAGS}) + target_link_libraries(parser_module_wasm_audit PRIVATE pj_base) + + file(GLOB _pj_parser_module_headers CONFIGURE_DEPENDS + "${CMAKE_CURRENT_SOURCE_DIR}/include/pj_base/parser_module/*.hpp") + set(_pj_wasm_dir "${CMAKE_CURRENT_BINARY_DIR}/parser_module_wasm_conformance") + set(_pj_wasm_raw "${_pj_wasm_dir}/toy_cdr_pointcloud.raw.wasm") + set(_pj_wasm_embedded "${_pj_wasm_dir}/toy_cdr_pointcloud.wasm") + set(_pj_wasm_source "${CMAKE_CURRENT_SOURCE_DIR}/tests/toy_cdr_pointcloud_module.cpp") + set(_pj_wasm_manifest "${CMAKE_CURRENT_SOURCE_DIR}/tests/toy_cdr_pointcloud.module.json") + + add_custom_command( + OUTPUT "${_pj_wasm_raw}" + COMMAND ${CMAKE_COMMAND} -E make_directory "${_pj_wasm_dir}" + COMMAND "${_pj_wasi_clang}" + --target=wasm32-wasip1 + --sysroot=${_pj_wasi_sysroot} + -mexec-model=reactor + -std=c++17 + -fno-exceptions + -fno-rtti + -fvisibility=hidden + -O1 + -Wall -Wextra -Werror + -DPJ_PARSER_MODULE_CLAIM_COUNT=2 + -I${CMAKE_CURRENT_SOURCE_DIR}/include + "${_pj_wasm_source}" + -Wl,--export=pj_module_abi + -Wl,--export=pj_module_create + -Wl,--export=pj_module_destroy + -Wl,--export=pj_module_bind + -Wl,--export=pj_module_parse + -Wl,--export=pj_module_last_error + -Wl,--export=pj_module_alloc + -Wl,--export=pj_module_free + -o "${_pj_wasm_raw}" + DEPENDS "${_pj_wasm_source}" ${_pj_parser_module_headers} + COMMENT "Compiling C++17 parser-module WASI reactor fixture" + VERBATIM + ) + add_custom_command( + OUTPUT "${_pj_wasm_embedded}" + COMMAND ${CMAKE_COMMAND} -E env ASAN_OPTIONS=detect_leaks=0 + $ + --embed "${_pj_wasm_raw}" "${_pj_wasm_manifest}" "${_pj_wasm_embedded}" + DEPENDS "${_pj_wasm_raw}" "${_pj_wasm_manifest}" parser_module_wasm_audit + COMMENT "Embedding parser-module wasm manifest with the shared codec" + VERBATIM + ) + add_custom_target(parser_module_wasm_conformance_fixture ALL + DEPENDS "${_pj_wasm_embedded}") + add_test(NAME parser_module_wasm_conformance_test + COMMAND parser_module_wasm_audit --audit "${_pj_wasm_embedded}" "${_pj_wasm_manifest}") + set_tests_properties(parser_module_wasm_conformance_test PROPERTIES + LABELS "parser_module;wasi" + REQUIRED_FILES "${_pj_wasm_embedded}") + endif() + + pj_add_parser_module(toy_cdr_pointcloud_module + SOURCE "${CMAKE_CURRENT_SOURCE_DIR}/tests/toy_cdr_pointcloud_module.cpp" + MANIFEST "${CMAKE_CURRENT_SOURCE_DIR}/tests/toy_cdr_pointcloud.module.json" + TARGETS native + ) endif() diff --git a/pj_base/include/pj_base/builtin_object_abi.h b/pj_base/include/pj_base/builtin_object_abi.h index 4fb253e4..af263f8f 100644 --- a/pj_base/include/pj_base/builtin_object_abi.h +++ b/pj_base/include/pj_base/builtin_object_abi.h @@ -8,10 +8,11 @@ * carrying a PJ_builtin_object_type_t. * * Canonical-object and pure-functional scalar production use the additive - * `pj.parser_functional.v1` C extension. MessageParserPluginBase keeps the - * plugin-author-facing ObjectRecord/ScalarRecord API inside the plugin DSO; - * its trampolines emit only POD scalar views or canonical wire bytes. The - * concrete host-owned C++ object is reconstructed on the host side. + * `pj.parser_functional.v1` and v2 C extensions. MessageParserPluginBase keeps + * the plugin-author-facing ObjectRecord/ScalarRecord API inside the plugin + * DSO; its trampolines emit only POD scalar views, canonical wire bytes, or a + * v2 splice using the frozen eligibility table below. The concrete host-owned + * C++ object is reconstructed on the host side. */ // Copyright 2026 Davide Faconti // SPDX-License-Identifier: Apache-2.0 @@ -74,6 +75,58 @@ typedef struct PJ_schema_classification_t { uint16_t reserved; } PJ_schema_classification_t; +/** One frozen canonical `PJ.*` bulk-field splice mapping. + * + * Object types appear only when their top-level canonical wire message has a + * single unambiguous bulk bytes field. New mappings append to the table; + * existing object-type/field-number pairs never change. + */ +typedef struct PJ_builtin_object_splice_field_v1_t { + uint16_t object_type; + uint16_t reserved; + uint32_t field_number; +} PJ_builtin_object_splice_field_v1_t; + +#define PJ_BUILTIN_OBJECT_SPLICE_FIELDS_V1_COUNT UINT32_C(9) + +/** Return the frozen splice-eligible table and optionally its entry count. */ +static inline const PJ_builtin_object_splice_field_v1_t* pj_builtin_object_splice_fields_v1(uint32_t* out_count) { + static const PJ_builtin_object_splice_field_v1_t fields[PJ_BUILTIN_OBJECT_SPLICE_FIELDS_V1_COUNT] = { + {PJ_BUILTIN_OBJECT_TYPE_IMAGE, 0, 7}, + {PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, 0, 9}, + {PJ_BUILTIN_OBJECT_TYPE_DEPTH_IMAGE, 0, 5}, + {PJ_BUILTIN_OBJECT_TYPE_OCCUPANCY_GRID, 0, 7}, + {PJ_BUILTIN_OBJECT_TYPE_COMPRESSED_POINTCLOUD, 0, 4}, + {PJ_BUILTIN_OBJECT_TYPE_MESH3D, 0, 7}, + {PJ_BUILTIN_OBJECT_TYPE_VIDEO_FRAME, 0, 3}, + {PJ_BUILTIN_OBJECT_TYPE_OCCUPANCY_GRID_UPDATE, 0, 7}, + {PJ_BUILTIN_OBJECT_TYPE_VOXEL_GRID, 0, 12}, + }; + if (out_count != NULL) { + *out_count = PJ_BUILTIN_OBJECT_SPLICE_FIELDS_V1_COUNT; + } + return fields; +} + +/** Look up an eligible field number. Returns false for ineligible types or a + * null output pointer. + */ +static inline bool pj_builtin_object_splice_field_number_v1(uint16_t object_type, uint32_t* out_field_number) { + uint32_t count = 0; + const PJ_builtin_object_splice_field_v1_t* fields = pj_builtin_object_splice_fields_v1(&count); + uint32_t index = 0; + if (out_field_number == NULL) { + return false; + } + for (index = 0; index < count; ++index) { + if (fields[index].object_type == object_type) { + *out_field_number = fields[index].field_number; + return true; + } + } + return false; +} + #ifdef __cplusplus } #endif diff --git a/pj_base/include/pj_base/message_parser_protocol.h b/pj_base/include/pj_base/message_parser_protocol.h index 937a0eb4..769dca6f 100644 --- a/pj_base/include/pj_base/message_parser_protocol.h +++ b/pj_base/include/pj_base/message_parser_protocol.h @@ -9,13 +9,14 @@ * per-record, with an optional append_arrow_stream tail slot for * parser-shaped formats that naturally decode batches. * - * Pure-functional production is the additive `pj.parser_functional.v1` - * extension declared in parser_functional_protocol.h. Scalar fields cross as - * synchronous borrowed C values; canonical objects cross as a stable type tag - * plus canonical wire bytes. No C++ parser class or STL value crosses the new - * path. Parsers built before SDK 0.21 do not expose the extension and may use a - * deprecated host compatibility bridge until the next SDK major version. - * Pure-C plugins may continue using parse() to write scalars to writeHost. + * Pure-functional production uses the additive `pj.parser_functional.v1` and + * v2 extensions declared in parser_functional_protocol.h. Scalar fields cross + * as synchronous borrowed C values; canonical objects cross as a stable type + * tag plus canonical wire bytes or a v2 splice. No C++ parser class or STL + * value crosses the new path. Parsers built before SDK 0.21 do not expose the + * extension and may use a deprecated host compatibility bridge until the next + * SDK major version. Pure-C plugins may continue using parse() to write + * scalars to writeHost. * * The host obtains the plugin's vtable via `PJ_get_message_parser_vtable()` * and drives the plugin through: create -> bind(registry) -> diff --git a/pj_base/include/pj_base/parser_functional_protocol.h b/pj_base/include/pj_base/parser_functional_protocol.h index 8f18bd6c..88849703 100644 --- a/pj_base/include/pj_base/parser_functional_protocol.h +++ b/pj_base/include/pj_base/parser_functional_protocol.h @@ -2,11 +2,11 @@ * @file parser_functional_protocol.h * @brief Additive C ABI extension for pure-functional MessageParser results. * - * A handler-based MessageParser built with SDK 0.21 or newer exposes this - * table from get_plugin_extension("pj.parser_functional.v1") after at least - * one SchemaHandler is registered. It replaces direct host calls on - * MessageParserPluginBase with synchronous, caller-owned C sinks. No C++ - * object, STL container, exception, or plugin allocation survives an + * A handler-based MessageParser exposes the v1 and v2 tables from + * get_plugin_extension after at least one SchemaHandler is registered. They + * replace direct host calls on MessageParserPluginBase with synchronous, + * caller-owned C sinks; v2 adds an optional single-field object splice. No + * C++ object, STL container, exception, or plugin allocation survives an * extension call. */ // Copyright 2026 Davide Faconti @@ -104,6 +104,58 @@ typedef struct PJ_parser_functional_v1_t { #define PJ_PARSER_FUNCTIONAL_V1_MIN_SIZE \ (offsetof(PJ_parser_functional_v1_t, parse_object) + sizeof(PJ_parser_parse_object_fn_t)) +#define PJ_PARSER_FUNCTIONAL_EXTENSION_V2 "pj.parser_functional.v2" +#define PJ_PARSER_ERROR_KIND_DATA_ERROR "pj.parser.data_error" +#define PJ_PARSER_ERROR_KIND_SINK_REJECTED "pj.parser.sink_rejected" +#define PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION "pj.parser.contract_violation" + +/** Caller-owned synchronous sink for one canonical object with one bulk + * field represented as a reference into the parse payload. + * + * `partial_wire` is canonical `PJ.*` wire with the eligible bulk field + * elided. `splice_field_number` identifies that field in the frozen table in + * builtin_object_abi.h. `input_offset` and `input_length` address bytes in + * the exact payload passed to parse_object. All views are borrowed for the + * callback duration. At most one splice may be emitted for a message. + */ +typedef bool (*PJ_parser_accept_object_spliced_fn_t)( + void* ctx, bool has_timestamp, int64_t timestamp_ns, uint16_t object_type, PJ_bytes_view_t partial_wire, + uint32_t splice_field_number, uint64_t input_offset, uint64_t input_length, PJ_error_t* out_error) PJ_NOEXCEPT; + +typedef struct PJ_parser_object_sink_v2_t { + /** Initialize to sizeof(PJ_parser_object_sink_v2_t). Future revisions may + * append fields; providers must accept every compatible prefix. + */ + uint32_t struct_size; + void* ctx; + PJ_parser_accept_object_fn_t accept_object; + PJ_parser_accept_object_spliced_fn_t accept_object_spliced; +} PJ_parser_object_sink_v2_t; + +#define PJ_PARSER_OBJECT_SINK_V2_MIN_SIZE \ + (offsetof(PJ_parser_object_sink_v2_t, accept_object_spliced) + sizeof(PJ_parser_accept_object_spliced_fn_t)) + +/** Pure-functional parser extension v2. + * + * Calls are [stream-thread] and synchronous. Scalar parsing is unchanged + * from v1. Object parsing consumes exactly one payload anchor reference using + * the v1 ownership contract and may deliver either a complete canonical wire + * object or one splice-eligible object. Failures set `extended_kind` to one + * of the frozen PJ_PARSER_ERROR_KIND_* values and leave `extended` null. + */ +typedef struct PJ_parser_functional_v2_t { + /** sizeof(PJ_parser_functional_v2_t) for this append-only table revision. */ + uint32_t struct_size; + PJ_parser_parse_scalars_fn_t parse_scalars; + bool (*parse_object)( + void* plugin_ctx, int64_t timestamp_ns, PJ_payload_t payload, const PJ_parser_object_sink_v2_t* sink, + PJ_error_t* out_error) PJ_NOEXCEPT; +} PJ_parser_functional_v2_t; + +#define PJ_PARSER_FUNCTIONAL_V2_MIN_SIZE \ + (offsetof(PJ_parser_functional_v2_t, parse_object) + \ + sizeof(bool (*)(void*, int64_t, PJ_payload_t, const PJ_parser_object_sink_v2_t*, PJ_error_t*) PJ_NOEXCEPT)) + #ifdef __cplusplus } #endif diff --git a/pj_base/include/pj_base/parser_module/README.md b/pj_base/include/pj_base/parser_module/README.md new file mode 100644 index 00000000..4ce87d68 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/README.md @@ -0,0 +1,58 @@ +# Functional parser-module authoring kit + +This directory is a standalone, header-only C++17 API. Headers in this subtree +may include only other headers from this subtree and C/C++ standard-library +headers. They must remain suitable for a WASI reactor build: no filesystem, +threads, iostreams, host SDK linkage, or exceptions crossing public/ABI calls. + +All data received from a host or message is borrowed through `ByteView` / +`PayloadView`. Fallible operations return the local `pj::Status` or +`pj::Expected` types. Output storage is owned by `pj::Blob`, whose allocation +entry points report failure instead of exposing allocation exceptions. + +The umbrella include for module authors is: + +```cpp +#include + +class RawImageParser final : public pj::FunctionalParser { + public: + pj::Status bind(const pj::BindingInfo& info) override { + return info.route() == pj::Route::kObject + ? pj::Status::ok() + : pj::Status::decline("object route only"); + } + + pj::Status parseObject(pj::PayloadView payload, pj::Timestamp timestamp, + pj::ObjectWriter& output) override { + auto image = output.image(); + if (timestamp.has_value) { + if (auto status = image.setTimestamp(timestamp.nanoseconds); + !status.isOk()) { + return status; + } + } + if (auto status = image.setEncoding("mono8"); !status.isOk()) { + return status; + } + return image.setData(payload); + } +}; + +PJ_FUNCTIONAL_PARSER(RawImageParser) +``` + +Override `parseScalars(PayloadView, Timestamp, ScalarWriter&)` for scalar +claims. `ObjectWriter` provides `image`, `pointCloud`, `depthImage`, +`occupancyGrid`, `compressedPointCloud`, `mesh3D`, `videoFrame`, +`occupancyGridUpdate`, and `voxelGrid` builders. + +Native modules are built with `pj_add_parser_module(... TARGETS native)`. The +target links no SDK library; it receives this subtree only as an include path. +`TARGETS wasm` is not available in SDK 0.22. The wasi-sdk gate compiles and +statically audits a reactor fixture, but it does not provide wasm authoring or +execution. + +See +`.claude/skills/plotjuggler-plugin/references/parser-module.md` at the repository +root for the manifest, schema-locator, splice, lifetime, and error contracts. diff --git a/pj_base/include/pj_base/parser_module/cdr_field_locator.hpp b/pj_base/include/pj_base/parser_module/cdr_field_locator.hpp new file mode 100644 index 00000000..46262540 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/cdr_field_locator.hpp @@ -0,0 +1,882 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file cdr_field_locator.hpp + * @brief Bind-time ROS 2 .msg bundle compiler and cached CDR field traversal. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/cdr_reader.hpp" + +namespace pj { +namespace detail { + +enum class CdrValueKind : uint8_t { + kBool, + kI8, + kU8, + kI16, + kU16, + kI32, + kU32, + kI64, + kU64, + kF32, + kF64, + kString, + kStruct, +}; + +enum class CdrContainerKind : uint8_t { + kScalar, + kFixedArray, + kSequence, +}; + +struct CdrSchemaType { + CdrValueKind kind = CdrValueKind::kU8; + CdrContainerKind container = CdrContainerKind::kScalar; + size_t fixed_count = 0; + size_t maximum_container_count = 0; + size_t maximum_string_length = 0; + std::string nested_name; + size_t nested_index = std::numeric_limits::max(); +}; + +struct CdrSchemaField { + std::string name; + CdrSchemaType type; +}; + +struct CdrSchemaStruct { + std::string name; + std::vector fields; +}; + +struct CdrRequestedField { + std::string path; + std::vector steps; + CdrReader::CachedKind cached_kind = CdrReader::CachedKind::kUnknown; +}; + +inline std::string_view trim(std::string_view text) { + while (!text.empty() && std::isspace(static_cast(text.front())) != 0) { + text.remove_prefix(1); + } + while (!text.empty() && std::isspace(static_cast(text.back())) != 0) { + text.remove_suffix(1); + } + return text; +} + +inline std::string normalizeRosType(std::string_view name) { + const size_t msg = name.find("/msg/"); + if (msg == std::string_view::npos) { + return std::string(name); + } + return std::string(name.substr(0, msg)) + "/" + std::string(name.substr(msg + 5)); +} + +inline Expected parseDecimal(std::string_view text) { + if (text.empty()) { + return Status::error("array extent is empty"); + } + size_t value = 0; + for (const char character : text) { + if (character < '0' || character > '9') { + return Status::error("array extent is not a decimal integer"); + } + const size_t digit = static_cast(character - '0'); + if (value > (std::numeric_limits::max() - digit) / 10) { + return Status::error("array extent overflows the host size type"); + } + value = value * 10 + digit; + } + return value; +} + +inline Expected parseCdrType(std::string_view spelling) { + CdrSchemaType type; + const size_t bracket = spelling.find('['); + std::string_view base = bracket == std::string_view::npos ? spelling : spelling.substr(0, bracket); + if (bracket != std::string_view::npos) { + if (spelling.back() != ']' || spelling.find('[', bracket + 1) != std::string_view::npos) { + return Status::error("unsupported ROS 2 array spelling"); + } + const std::string_view extent = spelling.substr(bracket + 1, spelling.size() - bracket - 2); + if (extent.empty()) { + type.container = CdrContainerKind::kSequence; + } else if (extent.substr(0, 2) == "<=") { + auto count = parseDecimal(extent.substr(2)); + if (!count || *count == 0) { + return Status::error("ROS 2 bounded sequences require a positive maximum"); + } + type.container = CdrContainerKind::kSequence; + type.maximum_container_count = *count; + } else { + auto count = parseDecimal(extent); + if (!count || *count == 0) { + return Status::error("ROS 2 fixed arrays require a positive extent"); + } + type.container = CdrContainerKind::kFixedArray; + type.fixed_count = *count; + } + } + if (base == "wstring" || base.substr(0, 9) == "wstring<=") { + return Status::error("wide ROS 2 strings are not supported by CdrFieldLocator"); + } + if (base.substr(0, 8) == "string<=") { + auto maximum = parseDecimal(base.substr(8)); + if (!maximum || *maximum == 0) { + return Status::error("bounded ROS 2 strings require a positive maximum"); + } + type.maximum_string_length = *maximum; + base = "string"; + } + + if (base == "bool") { + type.kind = CdrValueKind::kBool; + } else if (base == "int8") { + type.kind = CdrValueKind::kI8; + } else if (base == "uint8" || base == "byte" || base == "char") { + type.kind = CdrValueKind::kU8; + } else if (base == "int16") { + type.kind = CdrValueKind::kI16; + } else if (base == "uint16") { + type.kind = CdrValueKind::kU16; + } else if (base == "int32") { + type.kind = CdrValueKind::kI32; + } else if (base == "uint32") { + type.kind = CdrValueKind::kU32; + } else if (base == "int64") { + type.kind = CdrValueKind::kI64; + } else if (base == "uint64") { + type.kind = CdrValueKind::kU64; + } else if (base == "float32") { + type.kind = CdrValueKind::kF32; + } else if (base == "float64") { + type.kind = CdrValueKind::kF64; + } else if (base == "string") { + type.kind = CdrValueKind::kString; + } else { + type.kind = CdrValueKind::kStruct; + type.nested_name = normalizeRosType(base); + } + return type; +} + +inline CdrReader::CachedKind cachedKind(const CdrSchemaType& type) { + if (type.container == CdrContainerKind::kSequence || type.container == CdrContainerKind::kFixedArray) { + return type.kind == CdrValueKind::kU8 ? CdrReader::CachedKind::kBytes : CdrReader::CachedKind::kUnknown; + } + switch (type.kind) { + case CdrValueKind::kBool: + return CdrReader::CachedKind::kBool; + case CdrValueKind::kI8: + return CdrReader::CachedKind::kI8; + case CdrValueKind::kU8: + return CdrReader::CachedKind::kU8; + case CdrValueKind::kI16: + return CdrReader::CachedKind::kI16; + case CdrValueKind::kU16: + return CdrReader::CachedKind::kU16; + case CdrValueKind::kI32: + return CdrReader::CachedKind::kI32; + case CdrValueKind::kU32: + return CdrReader::CachedKind::kU32; + case CdrValueKind::kI64: + return CdrReader::CachedKind::kI64; + case CdrValueKind::kU64: + return CdrReader::CachedKind::kU64; + case CdrValueKind::kF32: + return CdrReader::CachedKind::kF32; + case CdrValueKind::kF64: + return CdrReader::CachedKind::kF64; + case CdrValueKind::kString: + return CdrReader::CachedKind::kString; + default: + return CdrReader::CachedKind::kUnknown; + } +} + +} // namespace detail + +class CdrTraversalPlan { + public: + CdrTraversalPlan() = default; + + [[nodiscard]] size_t size() const noexcept { + return requested_.size(); + } + + [[nodiscard]] Expected field(std::string_view path) const { + for (size_t index = 0; index < requested_.size(); ++index) { + if (requested_[index].path == path) { + return index; + } + } + return Status::error("field path is not present in the CDR traversal plan"); + } + + private: + [[nodiscard]] Status locateAll(CdrReader& reader) const { + if (!reader.status_.isOk()) { + return reader.status_; + } + std::array path{}; + Status result = walkStruct(reader, 0, 0, path); + if (!result.isOk()) { + return result; + } + for (const auto& cached : reader.cached_fields_) { + if (!cached.located) { + return reader.fail("requested CDR field was not reached during traversal"); + } + } + return Status::ok(); + } + + [[nodiscard]] Status walkStruct( + CdrReader& reader, size_t struct_index, size_t depth, + std::array& path) const { + if (depth >= CdrReader::kMaxTraversalDepth) { + return reader.fail("CDR traversal depth exceeds 64"); + } + const auto& structure = structs_[struct_index]; + for (size_t field_index = 0; field_index < structure.fields.size(); ++field_index) { + path[depth] = field_index; + size_t requested_id = std::numeric_limits::max(); + bool has_descendant = false; + for (size_t index = 0; index < requested_.size(); ++index) { + const auto& request = requested_[index]; + if (request.steps.size() <= depth) { + continue; + } + bool prefix_matches = true; + for (size_t step = 0; step <= depth; ++step) { + if (request.steps[step] != path[step]) { + prefix_matches = false; + break; + } + } + if (prefix_matches && request.steps.size() == depth + 1) { + requested_id = index; + } else if (prefix_matches) { + has_descendant = true; + } + } + Status result = walkValue(reader, structure.fields[field_index].type, depth, path, requested_id, has_descendant); + if (!result.isOk()) { + return result; + } + } + return Status::ok(); + } + + [[nodiscard]] Status walkValue( + CdrReader& reader, const detail::CdrSchemaType& type, size_t depth, + std::array& path, size_t requested_id, bool has_descendant) const { + if (has_descendant && + (type.kind != detail::CdrValueKind::kStruct || type.container != detail::CdrContainerKind::kScalar)) { + return reader.fail("CDR field path descends through a non-scalar nested struct"); + } + + if (type.container == detail::CdrContainerKind::kSequence) { + auto count = reader.readU32(); + if (!count) { + return count.status(); + } + if (type.maximum_container_count != 0 && *count > type.maximum_container_count) { + return reader.fail("CDR bounded-sequence length exceeds its schema maximum"); + } + if (type.kind == detail::CdrValueKind::kU8) { + if (static_cast(*count) > reader.remaining()) { + return reader.fail("CDR byte-sequence length exceeds the remaining payload"); + } + const size_t offset = reader.position_; + reader.position_ += *count; + if (requested_id != std::numeric_limits::max()) { + reader.cached_fields_[requested_id] = + CdrReader::CachedField{CdrReader::CachedKind::kBytes, offset, *count, true}; + } + return Status::ok(); + } + auto minimum = minimumSize(type, 0); + if (!minimum) { + return reader.fail(minimum.status().message()); + } + if (*count != 0 && *minimum == 0) { + return reader.fail("CDR sequence element has zero serialized minimum size"); + } + if (*minimum != 0 && static_cast(*count) > reader.remaining() / *minimum) { + return reader.fail("CDR sequence length exceeds the remaining payload"); + } + detail::CdrSchemaType element = type; + element.container = detail::CdrContainerKind::kScalar; + for (uint32_t index = 0; index < *count; ++index) { + const size_t before = reader.position_; + Status result = walkValue(reader, element, depth, path, std::numeric_limits::max(), false); + if (!result.isOk()) { + return result; + } + if (reader.position_ <= before) { + return reader.fail("CDR sequence element did not consume input"); + } + } + return Status::ok(); + } + + if (type.container == detail::CdrContainerKind::kFixedArray) { + if (type.kind == detail::CdrValueKind::kU8) { + if (type.fixed_count > reader.remaining()) { + return reader.fail("truncated CDR fixed byte array"); + } + const size_t offset = reader.position_; + reader.position_ += type.fixed_count; + if (requested_id != std::numeric_limits::max()) { + reader.cached_fields_[requested_id] = + CdrReader::CachedField{CdrReader::CachedKind::kBytes, offset, type.fixed_count, true}; + } + return Status::ok(); + } + detail::CdrSchemaType element = type; + element.container = detail::CdrContainerKind::kScalar; + auto minimum = minimumSize(element, 0); + if (!minimum) { + return reader.fail(minimum.status().message()); + } + if (type.fixed_count != 0 && *minimum == 0) { + return reader.fail("CDR fixed-array element has zero serialized minimum size"); + } + for (size_t index = 0; index < type.fixed_count; ++index) { + const size_t before = reader.position_; + Status result = walkValue(reader, element, depth, path, std::numeric_limits::max(), false); + if (!result.isOk()) { + return result; + } + if (reader.position_ <= before) { + return reader.fail("CDR fixed-array element did not consume input"); + } + } + return Status::ok(); + } + + if (type.kind == detail::CdrValueKind::kStruct) { + return walkStruct(reader, type.nested_index, depth + 1, path); + } + + if (type.kind == detail::CdrValueKind::kString) { + auto length = reader.readU32(); + if (!length) { + return length.status(); + } + if (*length == 0 || static_cast(*length) > reader.remaining()) { + return reader.fail("invalid CDR string length"); + } + if (type.maximum_string_length != 0 && static_cast(*length - 1) > type.maximum_string_length) { + return reader.fail("CDR bounded-string length exceeds its schema maximum"); + } + const size_t offset = reader.position_; + if (reader.payload_.data[offset + *length - 1] != 0) { + return reader.fail("CDR string is not NUL terminated"); + } + reader.position_ += *length; + if (requested_id != std::numeric_limits::max()) { + reader.cached_fields_[requested_id] = + CdrReader::CachedField{CdrReader::CachedKind::kString, offset, *length - 1, true}; + } + return Status::ok(); + } + + const size_t width = primitiveSize(type.kind); + Status aligned = reader.align(width); + if (!aligned.isOk()) { + return aligned; + } + if (width > reader.remaining()) { + return reader.fail("truncated CDR primitive while locating fields"); + } + const size_t offset = reader.position_; + if (type.kind == detail::CdrValueKind::kBool && reader.payload_.data[offset] > 1) { + return reader.fail("CDR bool is not 0 or 1"); + } + reader.position_ += width; + if (requested_id != std::numeric_limits::max()) { + reader.cached_fields_[requested_id] = CdrReader::CachedField{detail::cachedKind(type), offset, width, true}; + } + return Status::ok(); + } + + [[nodiscard]] Expected minimumSize(const detail::CdrSchemaType& type, size_t depth) const { + if (type.kind == detail::CdrValueKind::kString) { + return size_t{5}; + } + if (type.kind != detail::CdrValueKind::kStruct) { + return primitiveSize(type.kind); + } + if (depth >= CdrReader::kMaxTraversalDepth) { + return Status::error("CDR schema nesting depth exceeds 64"); + } + size_t total = 0; + for (const auto& field : structs_[type.nested_index].fields) { + auto minimum = minimumSize(field.type, depth + 1); + if (!minimum) { + return minimum.status(); + } + size_t field_minimum = *minimum; + if (field.type.container == detail::CdrContainerKind::kSequence) { + field_minimum = 4; + } else if (field.type.container == detail::CdrContainerKind::kFixedArray) { + if (field_minimum > std::numeric_limits::max() / field.type.fixed_count) { + return Status::error("CDR schema minimum size overflows the host size type"); + } + field_minimum *= field.type.fixed_count; + } + if (field_minimum > std::numeric_limits::max() - total) { + return Status::error("CDR schema minimum size overflows the host size type"); + } + total += field_minimum; + } + return total; + } + + [[nodiscard]] static size_t primitiveSize(detail::CdrValueKind kind) { + switch (kind) { + case detail::CdrValueKind::kBool: + case detail::CdrValueKind::kI8: + case detail::CdrValueKind::kU8: + return 1; + case detail::CdrValueKind::kI16: + case detail::CdrValueKind::kU16: + return 2; + case detail::CdrValueKind::kI32: + case detail::CdrValueKind::kU32: + case detail::CdrValueKind::kF32: + return 4; + case detail::CdrValueKind::kI64: + case detail::CdrValueKind::kU64: + case detail::CdrValueKind::kF64: + return 8; + default: + return 1; + } + } + + std::vector structs_; + std::vector requested_; + + friend class CdrFieldLocator; + friend class CdrReader; +}; + +class CdrFieldLocator { + public: + explicit CdrFieldLocator(std::string_view schema) { + PJ_PARSER_MODULE_TRY { + status_ = parse(schema); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + status_ = Status::error("allocation failed while compiling the ROS 2 schema"); + } + PJ_PARSER_MODULE_CATCH_ALL { + status_ = Status::error("unexpected failure while compiling the ROS 2 schema"); + } + } + + [[nodiscard]] const Status& status() const noexcept { + return status_; + } + + [[nodiscard]] Expected locate(std::initializer_list paths) const { + PJ_PARSER_MODULE_TRY { + return locate(std::vector(paths)); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while compiling CDR field paths"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while compiling CDR field paths"); + } + } + + [[nodiscard]] Expected locate(const std::vector& paths) const { + if (!status_.isOk()) { + return status_; + } + PJ_PARSER_MODULE_TRY { + CdrTraversalPlan plan; + plan.structs_ = structs_; + for (const auto& path : paths) { + auto request = compilePath(plan.structs_, path); + if (!request) { + return request.status(); + } + for (const auto& existing : plan.requested_) { + if (existing.path == request->path) { + return Status::error("duplicate CDR field path requested"); + } + } + plan.requested_.push_back(std::move(*request)); + } + if (plan.requested_.empty()) { + return Status::error("at least one CDR field path is required"); + } + return plan; + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while compiling CDR field paths"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while compiling CDR field paths"); + } + } + + private: + [[nodiscard]] Status parse(std::string_view schema) { + if (schema.empty()) { + return Status::error("ROS 2 .msg schema bundle is empty"); + } + structs_.clear(); + structs_.push_back(detail::CdrSchemaStruct{"", {}}); + size_t current = 0; + size_t position = 0; + while (position <= schema.size()) { + const size_t end = schema.find('\n', position); + std::string_view line = + schema.substr(position, end == std::string_view::npos ? schema.size() - position : end - position); + position = end == std::string_view::npos ? schema.size() + 1 : end + 1; + if (!line.empty() && line.back() == '\r') { + line.remove_suffix(1); + } + const size_t comment = line.find('#'); + if (comment != std::string_view::npos) { + line = line.substr(0, comment); + } + line = detail::trim(line); + if (line.empty() || line.find("===") == 0) { + continue; + } + if (line.find("---") == 0) { + return Status::error("ROS 2 service/action schemas are not supported by CdrFieldLocator"); + } + if (line.find("MSG:") == 0) { + const std::string name = detail::normalizeRosType(detail::trim(line.substr(4))); + if (name.empty()) { + return Status::error("ROS 2 schema bundle contains an empty MSG name"); + } + if (structs_.size() == 1 && structs_[0].fields.empty() && structs_[0].name.empty()) { + structs_[0].name = name; + current = 0; + } else { + for (const auto& structure : structs_) { + if (detail::normalizeRosType(structure.name) == name) { + return Status::error("ROS 2 schema bundle contains a duplicate MSG name"); + } + } + structs_.push_back(detail::CdrSchemaStruct{name, {}}); + current = structs_.size() - 1; + } + continue; + } + const size_t split = line.find_first_of(" \t"); + if (split == std::string_view::npos) { + return Status::error("malformed ROS 2 field declaration"); + } + const std::string_view type_spelling = line.substr(0, split); + std::string_view remainder = detail::trim(line.substr(split + 1)); + const size_t name_end = remainder.find_first_of(" \t="); + const std::string_view field_name = remainder.substr(0, name_end); + if (field_name.empty()) { + return Status::error("ROS 2 field declaration has no name"); + } + if (remainder.find('=') != std::string_view::npos) { + continue; + } + auto type = detail::parseCdrType(type_spelling); + if (!type) { + return type.status(); + } + for (const auto& field : structs_[current].fields) { + if (field.name == field_name) { + return Status::error("ROS 2 schema type contains a duplicate field name"); + } + } + structs_[current].fields.push_back(detail::CdrSchemaField{std::string(field_name), std::move(*type)}); + } + if (structs_[0].fields.empty()) { + return Status::error("ROS 2 .msg root type has no fields"); + } + for (auto& structure : structs_) { + for (auto& field : structure.fields) { + if (field.type.kind != detail::CdrValueKind::kStruct) { + continue; + } + auto nested = findStruct(field.type.nested_name); + if (!nested) { + return nested.status(); + } + field.type.nested_index = *nested; + } + } + std::vector visit_state(structs_.size(), 0); + for (size_t index = 0; index < structs_.size(); ++index) { + Status validated = validateSchemaGraph(index, visit_state, 0); + if (!validated.isOk()) { + return validated; + } + } + return Status::ok(); + } + + [[nodiscard]] Status validateSchemaGraph(size_t struct_index, std::vector& visit_state, size_t depth) const { + if (depth >= CdrReader::kMaxTraversalDepth) { + return Status::error("ROS 2 schema nesting depth exceeds 64"); + } + if (visit_state[struct_index] == 1) { + return Status::error("ROS 2 schema contains a cyclic nested type"); + } + if (visit_state[struct_index] == 2) { + return Status::ok(); + } + visit_state[struct_index] = 1; + for (const auto& field : structs_[struct_index].fields) { + if (field.type.kind != detail::CdrValueKind::kStruct) { + continue; + } + Status nested = validateSchemaGraph(field.type.nested_index, visit_state, depth + 1); + if (!nested.isOk()) { + return nested; + } + } + visit_state[struct_index] = 2; + return Status::ok(); + } + + [[nodiscard]] Expected findStruct(std::string_view name) const { + size_t match = std::numeric_limits::max(); + for (size_t index = 0; index < structs_.size(); ++index) { + const std::string normalized = detail::normalizeRosType(structs_[index].name); + const size_t slash = normalized.rfind('/'); + const std::string_view short_name = + slash == std::string::npos ? std::string_view(normalized) : std::string_view(normalized).substr(slash + 1); + if (normalized == name || short_name == name) { + if (match != std::numeric_limits::max()) { + return Status::error("ROS 2 nested type name is ambiguous in the schema bundle"); + } + match = index; + } + } + if (match == std::numeric_limits::max()) { + return Status::error("ROS 2 nested type is missing from the concatenated schema bundle"); + } + return match; + } + + [[nodiscard]] static Expected compilePath( + const std::vector& structs, std::string_view path) { + if (path.empty()) { + return Status::error("CDR field path is empty"); + } + detail::CdrRequestedField request; + request.path = std::string(path); + size_t struct_index = 0; + size_t position = 0; + while (position < path.size()) { + const size_t dot = path.find('.', position); + const std::string_view component = + path.substr(position, dot == std::string_view::npos ? path.size() - position : dot - position); + if (component.empty()) { + return Status::error("CDR field path contains an empty component"); + } + const auto& fields = structs[struct_index].fields; + size_t field_index = std::numeric_limits::max(); + for (size_t index = 0; index < fields.size(); ++index) { + if (fields[index].name == component) { + field_index = index; + break; + } + } + if (field_index == std::numeric_limits::max()) { + return Status::error("CDR field path is absent from the ROS 2 schema"); + } + request.steps.push_back(field_index); + const auto& type = fields[field_index].type; + if (dot == std::string_view::npos) { + request.cached_kind = detail::cachedKind(type); + if (request.cached_kind == CdrReader::CachedKind::kUnknown) { + return Status::error("requested CDR terminal field has an unsupported type"); + } + return request; + } + if (type.kind != detail::CdrValueKind::kStruct || type.container != detail::CdrContainerKind::kScalar) { + return Status::error("CDR field path may descend only through scalar nested structs"); + } + struct_index = type.nested_index; + position = dot + 1; + if (request.steps.size() >= CdrReader::kMaxTraversalDepth) { + return Status::error("CDR field path depth exceeds 64"); + } + } + return Status::error("CDR field path is malformed"); + } + + std::vector structs_; + Status status_; +}; + +inline CdrReader::CdrReader(PayloadView payload, const CdrTraversalPlan& plan) : payload_(payload), plan_(&plan) { + initialize(); + if (!status_.isOk()) { + return; + } + PJ_PARSER_MODULE_TRY { + cached_fields_.resize(plan.size()); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + status_ = Status::error("allocation failed while preparing CDR field cache"); + } + PJ_PARSER_MODULE_CATCH_ALL { + status_ = Status::error("unexpected failure while preparing CDR field cache"); + } +} + +inline Status CdrReader::ensurePlannedFields() { + if (!status_.isOk()) { + return status_; + } + if (plan_ == nullptr) { + return fail("CDR field access requires a traversal plan"); + } + if (traversal_count_ == 0) { + ++traversal_count_; + position_ = data_origin_; + depth_ = 0; + status_ = plan_->locateAll(*this); + } + return status_; +} + +inline Expected CdrReader::cached(CdrFieldId field, CachedKind expected) { + Status located = ensurePlannedFields(); + if (!located.isOk()) { + return located; + } + if (field >= cached_fields_.size()) { + return fail("CDR field id is outside the traversal plan"); + } + const CachedField value = cached_fields_[field]; + if (!value.located || value.kind != expected) { + return fail("CDR field accessor does not match the planned field type"); + } + return value; +} + +inline Expected CdrReader::cachedUnsigned(CdrFieldId field, CachedKind kind, size_t width) { + auto value = cached(field, kind); + if (!value) { + return value.status(); + } + uint64_t result = 0; + if (little_endian_) { + for (size_t index = 0; index < width; ++index) { + result |= static_cast(payload_.data[value->offset + index]) << (index * 8); + } + } else { + for (size_t index = 0; index < width; ++index) { + result = (result << 8) | payload_.data[value->offset + index]; + } + } + return result; +} + +inline Expected CdrReader::u32(CdrFieldId field) { + auto value = cachedUnsigned(field, CachedKind::kU32, 4); + return value ? Expected(static_cast(*value)) : Expected(value.status()); +} + +inline Expected CdrReader::boolean(CdrFieldId field) { + auto value = cachedUnsigned(field, CachedKind::kBool, 1); + if (!value) { + return value.status(); + } + if (*value > 1) { + return fail("CDR bool is not 0 or 1"); + } + return *value != 0; +} + +inline Expected CdrReader::i8(CdrFieldId field) { + return cachedBits(field, CachedKind::kI8); +} + +inline Expected CdrReader::u8(CdrFieldId field) { + auto value = cachedUnsigned(field, CachedKind::kU8, 1); + return value ? Expected(static_cast(*value)) : Expected(value.status()); +} + +inline Expected CdrReader::i16(CdrFieldId field) { + return cachedBits(field, CachedKind::kI16); +} + +inline Expected CdrReader::u16(CdrFieldId field) { + auto value = cachedUnsigned(field, CachedKind::kU16, 2); + return value ? Expected(static_cast(*value)) : Expected(value.status()); +} + +inline Expected CdrReader::i32(CdrFieldId field) { + return cachedBits(field, CachedKind::kI32); +} + +inline Expected CdrReader::u64(CdrFieldId field) { + return cachedUnsigned(field, CachedKind::kU64, 8); +} + +inline Expected CdrReader::i64(CdrFieldId field) { + return cachedBits(field, CachedKind::kI64); +} + +inline Expected CdrReader::f32(CdrFieldId field) { + return cachedBits(field, CachedKind::kF32); +} + +inline Expected CdrReader::f64(CdrFieldId field) { + return cachedBits(field, CachedKind::kF64); +} + +inline Expected CdrReader::string(CdrFieldId field) { + auto value = cached(field, CachedKind::kString); + if (!value) { + return value.status(); + } + return std::string_view(reinterpret_cast(payload_.data + value->offset), value->size); +} + +inline Expected CdrReader::bytes(CdrFieldId field) { + auto value = cached(field, CachedKind::kBytes); + if (!value) { + return value.status(); + } + return ByteView(payload_.data + value->offset, value->size); +} + +inline Expected CdrReader::spanRef(CdrFieldId field) { + auto value = cached(field, CachedKind::kBytes); + if (!value) { + return value.status(); + } + return InputSpanRef{static_cast(value->offset), static_cast(value->size)}; +} + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/cdr_reader.hpp b/pj_base/include/pj_base/parser_module/cdr_reader.hpp new file mode 100644 index 00000000..76d4aec4 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/cdr_reader.hpp @@ -0,0 +1,368 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file cdr_reader.hpp + * @brief Bounds-checked XCDR1 reader with optional compiled field traversal. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/core.hpp" + +namespace pj { + +class CdrTraversalPlan; +using CdrFieldId = size_t; + +class CdrReader { + public: + static constexpr size_t kMaxTraversalDepth = 64; + + explicit CdrReader(PayloadView payload) : payload_(payload) { + initialize(); + } + + CdrReader(PayloadView payload, const CdrTraversalPlan& plan); + + [[nodiscard]] const Status& status() const noexcept { + return status_; + } + + [[nodiscard]] bool littleEndian() const noexcept { + return little_endian_; + } + + [[nodiscard]] size_t position() const noexcept { + return position_; + } + + [[nodiscard]] size_t traversalCount() const noexcept { + return traversal_count_; + } + + [[nodiscard]] Status align(size_t alignment) { + if (!status_.isOk()) { + return status_; + } + if (alignment == 0 || (alignment & (alignment - 1)) != 0) { + return fail("CDR alignment must be a nonzero power of two"); + } + const size_t relative = position_ - data_origin_; + const size_t padding = (alignment - (relative & (alignment - 1))) & (alignment - 1); + if (padding > remaining()) { + return fail("truncated CDR alignment padding"); + } + position_ += padding; + return Status::ok(); + } + + [[nodiscard]] Expected readBool() { + auto value = readU8(); + if (!value) { + return value.status(); + } + if (*value > 1) { + return fail("CDR bool is not 0 or 1"); + } + return *value != 0; + } + + [[nodiscard]] Expected readI8() { + return readSigned(); + } + + [[nodiscard]] Expected readU8() { + return readUnsigned(); + } + + [[nodiscard]] Expected readI16() { + return readSigned(); + } + + [[nodiscard]] Expected readU16() { + return readUnsigned(); + } + + [[nodiscard]] Expected readI32() { + return readSigned(); + } + + [[nodiscard]] Expected readU32() { + return readUnsigned(); + } + + [[nodiscard]] Expected readI64() { + return readSigned(); + } + + [[nodiscard]] Expected readU64() { + return readUnsigned(); + } + + [[nodiscard]] Expected readF32() { + auto bits = readU32(); + if (!bits) { + return bits.status(); + } + float value = 0; + const uint32_t raw = *bits; + std::memcpy(&value, &raw, sizeof(value)); + return value; + } + + [[nodiscard]] Expected readF64() { + auto bits = readU64(); + if (!bits) { + return bits.status(); + } + double value = 0; + const uint64_t raw = *bits; + std::memcpy(&value, &raw, sizeof(value)); + return value; + } + + [[nodiscard]] Expected readString() { + auto length = readU32(); + if (!length) { + return length.status(); + } + if (*length == 0) { + return fail("CDR string length does not include a terminator"); + } + if (static_cast(*length) > remaining()) { + return fail("CDR string length exceeds the remaining payload"); + } + const size_t size = *length; + const char* chars = reinterpret_cast(payload_.data + position_); + if (chars[size - 1] != '\0') { + return fail("CDR string is not NUL terminated"); + } + position_ += size; + return std::string_view(chars, size - 1); + } + + [[nodiscard]] Expected readSequenceLength(size_t minimum_element_size = 1) { + auto count = readU32(); + if (!count) { + return count.status(); + } + if (minimum_element_size == 0) { + return fail("CDR sequence element size must be nonzero"); + } + if (static_cast(*count) > remaining() / minimum_element_size) { + return fail("CDR sequence length exceeds the remaining payload"); + } + return *count; + } + + [[nodiscard]] Expected readByteSequence() { + auto count = readSequenceLength(1); + if (!count) { + return count.status(); + } + const size_t size = *count; + ByteView result(payload_.data + position_, size); + position_ += size; + return result; + } + + template + [[nodiscard]] Status readFixedArray(std::array& output) { + for (auto& value : output) { + auto next = readPrimitive(); + if (!next) { + return next.status(); + } + value = *next; + } + return Status::ok(); + } + + [[nodiscard]] Status enterStruct() { + if (depth_ == kMaxTraversalDepth) { + return fail("CDR traversal depth exceeds 64"); + } + ++depth_; + return Status::ok(); + } + + [[nodiscard]] Status leaveStruct() { + if (depth_ == 0) { + return fail("CDR struct-depth underflow"); + } + --depth_; + return Status::ok(); + } + + [[nodiscard]] Expected u32(CdrFieldId field); + [[nodiscard]] Expected boolean(CdrFieldId field); + [[nodiscard]] Expected i8(CdrFieldId field); + [[nodiscard]] Expected u8(CdrFieldId field); + [[nodiscard]] Expected i16(CdrFieldId field); + [[nodiscard]] Expected u16(CdrFieldId field); + [[nodiscard]] Expected i32(CdrFieldId field); + [[nodiscard]] Expected u64(CdrFieldId field); + [[nodiscard]] Expected i64(CdrFieldId field); + [[nodiscard]] Expected f32(CdrFieldId field); + [[nodiscard]] Expected f64(CdrFieldId field); + [[nodiscard]] Expected string(CdrFieldId field); + [[nodiscard]] Expected bytes(CdrFieldId field); + [[nodiscard]] Expected spanRef(CdrFieldId field); + + /// Internal cache vocabulary exposed only so the header-only traversal plan + /// can remain a separate type without dynamic polymorphism. + enum class CachedKind : uint8_t { + kUnknown, + kBool, + kI8, + kU8, + kI16, + kU16, + kI32, + kU32, + kI64, + kU64, + kF32, + kF64, + kString, + kBytes, + }; + + struct CachedField { + CachedKind kind = CachedKind::kUnknown; + size_t offset = 0; + size_t size = 0; + bool located = false; + }; + + private: + void initialize() { + if (payload_.data == nullptr || payload_.size < 4) { + status_ = Status::error("truncated CDR encapsulation header"); + return; + } + const uint16_t representation = + static_cast((static_cast(payload_.data[0]) << 8) | payload_.data[1]); + if (representation != 0 && representation != 1) { + status_ = Status::error("unsupported CDR representation; XCDR1 plain CDR is required"); + return; + } + little_endian_ = representation == 1; + data_origin_ = 4; + position_ = data_origin_; + } + + [[nodiscard]] size_t remaining() const noexcept { + return position_ <= payload_.size ? payload_.size - position_ : 0; + } + + [[nodiscard]] Status fail(std::string_view message) noexcept { + status_ = Status::error(message); + return status_; + } + + template + [[nodiscard]] Expected readUnsigned() { + static_assert(std::is_unsigned::value, "UInt must be unsigned"); + Status aligned = align(sizeof(UInt)); + if (!aligned.isOk()) { + return aligned; + } + if (sizeof(UInt) > remaining()) { + return fail("truncated CDR primitive"); + } + UInt value = 0; + if (little_endian_) { + for (size_t index = 0; index < sizeof(UInt); ++index) { + value |= static_cast(payload_.data[position_ + index]) << (index * 8); + } + } else { + for (size_t index = 0; index < sizeof(UInt); ++index) { + value = static_cast((value << 8) | payload_.data[position_ + index]); + } + } + position_ += sizeof(UInt); + return value; + } + + template + [[nodiscard]] Expected readSigned() { + auto bits = readUnsigned(); + if (!bits) { + return bits.status(); + } + Signed value = 0; + const Unsigned raw = *bits; + std::memcpy(&value, &raw, sizeof(value)); + return value; + } + + template + [[nodiscard]] Expected readPrimitive() { + if constexpr (std::is_same::value) { + return readBool(); + } else if constexpr (std::is_same::value) { + return readI8(); + } else if constexpr (std::is_same::value) { + return readU8(); + } else if constexpr (std::is_same::value) { + return readI16(); + } else if constexpr (std::is_same::value) { + return readU16(); + } else if constexpr (std::is_same::value) { + return readI32(); + } else if constexpr (std::is_same::value) { + return readU32(); + } else if constexpr (std::is_same::value) { + return readI64(); + } else if constexpr (std::is_same::value) { + return readU64(); + } else if constexpr (std::is_same::value) { + return readF32(); + } else if constexpr (std::is_same::value) { + return readF64(); + } else { + static_assert(!std::is_same::value, "unsupported CDR primitive type"); + } + } + + [[nodiscard]] Status ensurePlannedFields(); + [[nodiscard]] Expected cached(CdrFieldId field, CachedKind expected); + [[nodiscard]] Expected cachedUnsigned(CdrFieldId field, CachedKind kind, size_t width); + + /// Reinterpret a cached unsigned field of width sizeof(Bits) as Value. The + /// cached-field counterpart of readSigned(). + template + [[nodiscard]] Expected cachedBits(CdrFieldId field, CachedKind kind) { + auto value = cachedUnsigned(field, kind, sizeof(Bits)); + if (!value) { + return value.status(); + } + const Bits bits = static_cast(*value); + Value result = 0; + std::memcpy(&result, &bits, sizeof(result)); + return result; + } + + PayloadView payload_; + Status status_; + bool little_endian_ = false; + size_t data_origin_ = 0; + size_t position_ = 0; + size_t depth_ = 0; + const CdrTraversalPlan* plan_ = nullptr; + std::vector cached_fields_; + size_t traversal_count_ = 0; + + friend class CdrTraversalPlan; +}; + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/core.hpp b/pj_base/include/pj_base/parser_module/core.hpp new file mode 100644 index 00000000..a9e8ff8e --- /dev/null +++ b/pj_base/include/pj_base/parser_module/core.hpp @@ -0,0 +1,424 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file core.hpp + * @brief Standalone value, view, and allocation vocabulary for parser modules. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if defined(_MSVC_LANG) +static_assert(_MSVC_LANG >= 201703L, "pj parser modules require C++17 or newer"); +#else +static_assert(__cplusplus >= 201703L, "pj parser modules require C++17 or newer"); +#endif + +// Keep the same source valid for native exception-enabled builds and WASI +// reactor builds compiled with -fno-exceptions. Public operations return +// Status in both modes; no exception syntax reaches the latter compiler. +#if defined(__cpp_exceptions) || defined(_CPPUNWIND) +#define PJ_PARSER_MODULE_TRY try +#define PJ_PARSER_MODULE_CATCH_BAD_ALLOC catch (const std::bad_alloc&) +#define PJ_PARSER_MODULE_CATCH_ALL catch (...) +#else +#define PJ_PARSER_MODULE_TRY if (true) +#define PJ_PARSER_MODULE_CATCH_BAD_ALLOC else if (false) +#define PJ_PARSER_MODULE_CATCH_ALL else +#endif + +namespace pj { + +enum class StatusCode : uint8_t { + kOk, + kDecline, + kError, +}; + +class Status { + public: + static constexpr size_t kMessageCapacity = 512; + + Status() = default; + + [[nodiscard]] static Status ok() noexcept { + return {}; + } + + [[nodiscard]] static Status decline(std::string_view message) noexcept { + return Status(StatusCode::kDecline, message); + } + + [[nodiscard]] static Status error(std::string_view message) noexcept { + return Status(StatusCode::kError, message); + } + + [[nodiscard]] bool isOk() const noexcept { + return code_ == StatusCode::kOk; + } + + [[nodiscard]] bool isDecline() const noexcept { + return code_ == StatusCode::kDecline; + } + + [[nodiscard]] bool isError() const noexcept { + return code_ == StatusCode::kError; + } + + [[nodiscard]] StatusCode code() const noexcept { + return code_; + } + + [[nodiscard]] std::string_view message() const noexcept { + return {message_.data(), message_size_}; + } + + private: + Status(StatusCode code, std::string_view message) noexcept : code_(code) { + message_size_ = message.size() < message_.size() - 1 ? message.size() : message_.size() - 1; + if (message_size_ != 0) { + std::memcpy(message_.data(), message.data(), message_size_); + } + message_[message_size_] = '\0'; + } + + StatusCode code_ = StatusCode::kOk; + std::array message_{}; + size_t message_size_ = 0; +}; + +template +class Expected { + public: + Expected(const T& value) : storage_(value) {} + Expected(T&& value) : storage_(std::move(value)) {} + Expected(Status error) : storage_(std::move(error)) {} + + [[nodiscard]] bool hasValue() const noexcept { + return std::holds_alternative(storage_); + } + + [[nodiscard]] explicit operator bool() const noexcept { + return hasValue(); + } + + [[nodiscard]] T& value() & { + return std::get(storage_); + } + + [[nodiscard]] const T& value() const& { + return std::get(storage_); + } + + [[nodiscard]] T&& value() && { + return std::get(std::move(storage_)); + } + + [[nodiscard]] T* operator->() { + return &value(); + } + + [[nodiscard]] const T* operator->() const { + return &value(); + } + + [[nodiscard]] T& operator*() & { + return value(); + } + + [[nodiscard]] const T& operator*() const& { + return value(); + } + + [[nodiscard]] Status status() const { + return hasValue() ? Status::ok() : std::get(storage_); + } + + private: + std::variant storage_; +}; + +struct ByteView { + const uint8_t* data = nullptr; + size_t size = 0; + + ByteView() = default; + ByteView(const uint8_t* input_data, size_t input_size) : data(input_data), size(input_size) {} + + template + ByteView(const uint8_t (&bytes)[Size]) : data(bytes), size(Size) {} + + [[nodiscard]] bool empty() const noexcept { + return size == 0; + } + + [[nodiscard]] Expected subview(size_t offset, size_t length) const { + if (data == nullptr && size != 0) { + return Status::error("byte-view storage is null"); + } + if (offset > size || length > size - offset) { + return Status::error("byte-view range is outside its storage"); + } + return ByteView(data == nullptr ? nullptr : data + offset, length); + } +}; + +using PayloadView = ByteView; + +struct MutableByteView { + uint8_t* data = nullptr; + size_t size = 0; +}; + +struct InputSpanRef { + uint64_t offset = 0; + uint64_t length = 0; +}; + +class Blob { + public: + Blob() = default; + ~Blob() { + delete[] data_; + } + + Blob(Blob&& other) noexcept + : data_(std::exchange(other.data_, nullptr)), + size_(std::exchange(other.size_, 0)), + capacity_(std::exchange(other.capacity_, 0)) {} + + Blob& operator=(Blob&& other) noexcept { + if (this != &other) { + delete[] data_; + data_ = std::exchange(other.data_, nullptr); + size_ = std::exchange(other.size_, 0); + capacity_ = std::exchange(other.capacity_, 0); + } + return *this; + } + + Blob(const Blob&) = delete; + Blob& operator=(const Blob&) = delete; + + [[nodiscard]] const uint8_t* data() const noexcept { + return data_; + } + + [[nodiscard]] uint8_t* data() noexcept { + return data_; + } + + [[nodiscard]] size_t size() const noexcept { + return size_; + } + + [[nodiscard]] bool empty() const noexcept { + return size_ == 0; + } + + [[nodiscard]] ByteView view() const noexcept { + return {data_, size_}; + } + + [[nodiscard]] ByteView bytes() const noexcept { + return view(); + } + + [[nodiscard]] Status reserve(size_t size) noexcept { + if (size <= capacity_) { + return Status::ok(); + } + auto* grown = new (std::nothrow) uint8_t[size]; + if (grown == nullptr) { + return Status::error("parser-module allocation failed"); + } + if (size_ != 0) { + std::memcpy(grown, data_, size_); + } + delete[] data_; + data_ = grown; + capacity_ = size; + return Status::ok(); + } + + [[nodiscard]] Status resize(size_t size) noexcept { + Status reserved = reserve(size); + if (!reserved.isOk()) { + return reserved; + } + if (size > size_) { + std::memset(data_ + size_, 0, size - size_); + } + size_ = size; + return Status::ok(); + } + + [[nodiscard]] Status append(ByteView bytes) noexcept { + if (bytes.size != 0 && bytes.data == nullptr) { + return Status::error("cannot append a null non-empty byte view"); + } + if (bytes.size == 0) { + return Status::ok(); + } + if (bytes.size > std::numeric_limits::max() - size_) { + return Status::error("parser-module output size overflow"); + } + const size_t required = size_ + bytes.size; + if (required > capacity_) { + size_t next_capacity = capacity_ == 0 ? size_t{64} : capacity_; + while (next_capacity < required) { + if (next_capacity > std::numeric_limits::max() / 2) { + next_capacity = required; + break; + } + next_capacity *= 2; + } + Status reserved = reserve(next_capacity); + if (!reserved.isOk()) { + return reserved; + } + } + std::memcpy(data_ + size_, bytes.data, bytes.size); + size_ = required; + return Status::ok(); + } + + [[nodiscard]] Status push(uint8_t byte) noexcept { + if (size_ == capacity_) { + const size_t next_capacity = capacity_ == 0 ? size_t{64} : capacity_ * 2; + if (next_capacity < capacity_) { + return Status::error("parser-module output size overflow"); + } + Status reserved = reserve(next_capacity); + if (!reserved.isOk()) { + return reserved; + } + } + data_[size_++] = byte; + return Status::ok(); + } + + private: + uint8_t* data_ = nullptr; + size_t size_ = 0; + size_t capacity_ = 0; +}; + +[[nodiscard]] inline Expected allocateBlob(size_t size) noexcept { + Blob blob; + Status status = blob.resize(size); + if (!status.isOk()) { + return status; + } + return blob; +} + +/// Realloc-backed monotonic storage for bind-time plans and temporary output. +/// Every growth reports allocation failure; individual allocations are freed +/// together when the arena is destroyed or reset. A growth may relocate the +/// arena, so previously returned views are valid only until the next allocate. +class BumpArena { + public: + BumpArena() = default; + ~BumpArena() { + std::free(data_); + } + + BumpArena(const BumpArena&) = delete; + BumpArena& operator=(const BumpArena&) = delete; + + BumpArena(BumpArena&& other) noexcept + : data_(std::exchange(other.data_, nullptr)), + capacity_(std::exchange(other.capacity_, 0)), + used_(std::exchange(other.used_, 0)) {} + + BumpArena& operator=(BumpArena&& other) noexcept { + if (this != &other) { + std::free(data_); + data_ = std::exchange(other.data_, nullptr); + capacity_ = std::exchange(other.capacity_, 0); + used_ = std::exchange(other.used_, 0); + } + return *this; + } + + [[nodiscard]] Expected allocate(size_t size, size_t alignment = alignof(std::max_align_t)) noexcept { + if (alignment == 0 || (alignment & (alignment - 1)) != 0) { + return Status::error("arena alignment must be a nonzero power of two"); + } + if (alignment > alignof(std::max_align_t)) { + return Status::error("arena alignment exceeds malloc alignment"); + } + if (size == 0) { + return MutableByteView{}; + } + const size_t padding = (alignment - (used_ & (alignment - 1))) & (alignment - 1); + if (padding > std::numeric_limits::max() - used_ || + size > std::numeric_limits::max() - used_ - padding) { + return Status::error("arena allocation size overflow"); + } + const size_t required = used_ + padding + size; + if (required > capacity_) { + size_t next_capacity = capacity_ == 0 ? size_t{256} : capacity_; + while (next_capacity < required) { + if (next_capacity > std::numeric_limits::max() / 2) { + next_capacity = required; + break; + } + next_capacity *= 2; + } + void* grown = std::realloc(data_, next_capacity); + if (grown == nullptr) { + return Status::error("arena allocation failed"); + } + data_ = static_cast(grown); + capacity_ = next_capacity; + } + used_ += padding; + MutableByteView result{data_ + used_, size}; + used_ += size; + return result; + } + + void reset() noexcept { + used_ = 0; + } + + [[nodiscard]] size_t used() const noexcept { + return used_; + } + + private: + uint8_t* data_ = nullptr; + size_t capacity_ = 0; + size_t used_ = 0; +}; + +namespace detail { + +inline void copyMessage(char* destination, size_t capacity, std::string_view message) noexcept { + if (destination == nullptr || capacity == 0) { + return; + } + const size_t length = message.size() < capacity - 1 ? message.size() : capacity - 1; + if (length != 0) { + std::memcpy(destination, message.data(), length); + } + destination[length] = '\0'; +} + +} // namespace detail +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/module.hpp b/pj_base/include/pj_base/parser_module/module.hpp new file mode 100644 index 00000000..40d775b5 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/module.hpp @@ -0,0 +1,733 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file module.hpp + * @brief Umbrella authoring API and complete functional-module export macro. + */ + +#include +#if !defined(__wasm__) +#include +#endif +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/cdr_field_locator.hpp" +#include "pj_base/parser_module/cdr_reader.hpp" +#include "pj_base/parser_module/core.hpp" +#include "pj_base/parser_module/object_writer.hpp" +#include "pj_base/parser_module/proto_field_locator.hpp" +#include "pj_base/parser_module/proto_reader.hpp" +#include "pj_base/parser_module/time.hpp" + +#if defined(PJ_PARSER_MODULE_MANIFEST_HEADER) +#include PJ_PARSER_MODULE_MANIFEST_HEADER +#endif + +#if !defined(PJ_PARSER_MODULE_CLAIM_COUNT) +#define PJ_PARSER_MODULE_CLAIM_COUNT 0 +#endif + +#if !defined(PJ_PARSER_MODULE_HAS_MANIFEST) +namespace pj { +namespace detail { +inline constexpr char kBuiltManifest[] = "{}"; +} +} // namespace pj +#endif + +#if defined(_WIN32) +#define PJ_PARSER_MODULE_EXPORT __declspec(dllexport) +#else +#define PJ_PARSER_MODULE_EXPORT __attribute__((visibility("default"))) +#endif + +namespace pj { + +inline constexpr uint32_t kModuleAbiVersion = 1; +inline constexpr int32_t kModuleOk = 0; +inline constexpr int32_t kModuleDecline = 1; +inline constexpr int32_t kModuleError = -1; +inline constexpr int32_t kModuleBadToken = -2; +inline constexpr int32_t kModuleMalformedInput = -3; +inline constexpr int32_t kModuleBadClaimIndex = -4; +inline constexpr int32_t kModuleAllocationFailure = -5; +inline constexpr uint64_t kCreationErrorToken = 0; +inline constexpr size_t kErrorBufferSize = 512; + +enum class Route : uint16_t { + kScalar = 1, + kObject = 2, +}; + +/// Optional timestamp supplied with one parse input. This is a per-message +/// value; it is never retained in BindingInfo. +struct Timestamp { + bool has_value = false; + int64_t nanoseconds = 0; +}; + +class OwnedBindingInfo { + public: + [[nodiscard]] Route route() const noexcept { + return route_; + } + [[nodiscard]] uint32_t claimIndex() const noexcept { + return claim_index_; + } + [[nodiscard]] uint16_t expectedObjectType() const noexcept { + return expected_object_type_; + } + [[nodiscard]] ByteView encoding() const noexcept { + return encoding_.view(); + } + [[nodiscard]] ByteView typeName() const noexcept { + return type_name_.view(); + } + [[nodiscard]] ByteView schema() const noexcept { + return schema_.view(); + } + [[nodiscard]] ByteView claimId() const noexcept { + return claim_id_.view(); + } + [[nodiscard]] ByteView configJson() const noexcept { + return config_json_.view(); + } + [[nodiscard]] ByteView schemaDigest() const noexcept { + return schema_digest_.view(); + } + [[nodiscard]] std::string_view schemaText() const noexcept { + return std::string_view(reinterpret_cast(schema_.data()), schema_.size()); + } + + private: + Route route_ = Route::kScalar; + uint32_t claim_index_ = 0; + uint16_t expected_object_type_ = 0; + Blob encoding_; + Blob type_name_; + Blob schema_; + Blob claim_id_; + Blob config_json_; + Blob schema_digest_; + + friend class BindingInfo; +}; + +/// Borrowed view over one bind call's encoded block. Every returned ByteView +/// expires when bind() returns. Modules that retain metadata use owningCopy(). +class BindingInfo { + public: + [[nodiscard]] Route route() const noexcept { + return route_; + } + [[nodiscard]] uint32_t claimIndex() const noexcept { + return claim_index_; + } + [[nodiscard]] uint16_t expectedObjectType() const noexcept { + return expected_object_type_; + } + [[nodiscard]] ByteView encoding() const noexcept { + return encoding_; + } + [[nodiscard]] ByteView typeName() const noexcept { + return type_name_; + } + [[nodiscard]] ByteView schema() const noexcept { + return schema_; + } + [[nodiscard]] ByteView claimId() const noexcept { + return claim_id_; + } + [[nodiscard]] ByteView configJson() const noexcept { + return config_json_; + } + [[nodiscard]] ByteView schemaDigest() const noexcept { + return schema_digest_; + } + [[nodiscard]] std::string_view schemaText() const noexcept { + return std::string_view(reinterpret_cast(schema_.data), schema_.size); + } + + [[nodiscard]] Expected owningCopy() const noexcept { + OwnedBindingInfo owned; + owned.route_ = route_; + owned.claim_index_ = claim_index_; + owned.expected_object_type_ = expected_object_type_; + const auto copy = [](Blob& destination, ByteView source) { + Status resized = destination.resize(source.size); + if (!resized.isOk()) { + return resized; + } + if (source.size != 0) { + if (source.data == nullptr) { + return Status::error("BindingInfo field has null storage"); + } + std::memcpy(destination.data(), source.data, source.size); + } + return Status::ok(); + }; + for (const auto& field : { + std::pair{&owned.encoding_, encoding_}, + {&owned.type_name_, type_name_}, + {&owned.schema_, schema_}, + {&owned.claim_id_, claim_id_}, + {&owned.config_json_, config_json_}, + {&owned.schema_digest_, schema_digest_}, + }) { + Status copied = copy(*field.first, field.second); + if (!copied.isOk()) { + return copied; + } + } + return owned; + } + + private: + Route route_ = Route::kScalar; + uint32_t claim_index_ = 0; + uint16_t expected_object_type_ = 0; + ByteView encoding_; + ByteView type_name_; + ByteView schema_; + ByteView claim_id_; + ByteView config_json_; + ByteView schema_digest_; + + friend Expected readBindingInfo(ByteView); +}; + +struct ParseInput { + bool has_timestamp = false; + int64_t timestamp_ns = 0; + PayloadView payload; +}; + +namespace detail { + +class LittleEndianReader { + public: + explicit LittleEndianReader(ByteView bytes) : bytes_(bytes) {} + + template + [[nodiscard]] bool read(UInt& value) { + static_assert(std::is_unsigned::value, "little-endian values must be unsigned"); + if (position_ > bytes_.size || sizeof(UInt) > bytes_.size - position_) { + return false; + } + value = 0; + for (size_t index = 0; index < sizeof(UInt); ++index) { + value |= static_cast(bytes_.data[position_ + index]) << (index * 8U); + } + position_ += sizeof(UInt); + return true; + } + + [[nodiscard]] bool skip(size_t size) { + if (position_ > bytes_.size || size > bytes_.size - position_) { + return false; + } + position_ += size; + return true; + } + + [[nodiscard]] size_t position() const noexcept { + return position_; + } + + private: + ByteView bytes_; + size_t position_ = 0; +}; + +inline uint64_t addressOf(const void* pointer) noexcept { + return static_cast(reinterpret_cast(pointer)); +} + +inline void recordError(std::array& destination, std::string_view message) noexcept { + copyMessage(destination.data(), destination.size(), message); +} + +} // namespace detail + +[[nodiscard]] inline Expected readBindingInfo(ByteView bytes) { + if (bytes.data == nullptr || bytes.size < 16) { + return Status::error("truncated BindingInfo v1 header"); + } + detail::LittleEndianReader reader(bytes); + uint16_t version = 0; + uint16_t route = 0; + uint32_t claim_index = 0; + uint16_t expected_type = 0; + uint16_t reserved = 0; + uint32_t field_count = 0; + if (!reader.read(version) || !reader.read(route) || !reader.read(claim_index) || !reader.read(expected_type) || + !reader.read(reserved) || !reader.read(field_count)) { + return Status::error("truncated BindingInfo v1 header"); + } + (void)reserved; + if (version != 1 || (route != 1 && route != 2)) { + return Status::error("unsupported BindingInfo version or route"); + } + if (field_count < 6 || static_cast(field_count) > (bytes.size - reader.position()) / 8) { + return Status::error("truncated BindingInfo field table"); + } + std::array fields{}; + for (uint32_t index = 0; index < field_count; ++index) { + uint32_t offset = 0; + uint32_t length = 0; + if (!reader.read(offset) || !reader.read(length)) { + return Status::error("truncated BindingInfo field descriptor"); + } + if (static_cast(offset) > bytes.size || static_cast(length) > bytes.size - offset) { + return Status::error("BindingInfo field range is outside the block"); + } + if (index < fields.size()) { + fields[index] = ByteView(bytes.data + offset, length); + } + } + BindingInfo info; + info.route_ = static_cast(route); + info.claim_index_ = claim_index; + info.expected_object_type_ = expected_type; + info.encoding_ = fields[0]; + info.type_name_ = fields[1]; + info.schema_ = fields[2]; + info.claim_id_ = fields[3]; + info.config_json_ = fields[4]; + info.schema_digest_ = fields[5]; + return info; +} + +[[nodiscard]] inline Expected readParseInput(ByteView bytes) { + if (bytes.data == nullptr || bytes.size < 24) { + return Status::error("truncated ParseInput v1 header"); + } + const uint8_t flags = bytes.data[0]; + if ((flags & UINT8_C(0xFE)) != 0) { + return Status::error("ParseInput v1 has unknown flag bits"); + } + detail::LittleEndianReader reader(bytes); + if (!reader.skip(8)) { + return Status::error("truncated ParseInput v1 flags"); + } + uint64_t timestamp_bits = 0; + uint64_t payload_length = 0; + if (!reader.read(timestamp_bits) || !reader.read(payload_length)) { + return Status::error("truncated ParseInput v1 header"); + } + if (payload_length > std::numeric_limits::max() || payload_length != bytes.size - reader.position()) { + return Status::error("ParseInput payload length does not match the block"); + } + int64_t timestamp = 0; + std::memcpy(×tamp, ×tamp_bits, sizeof(timestamp)); + ParseInput input; + input.has_timestamp = (flags & 1U) != 0; + input.timestamp_ns = timestamp; + input.payload = PayloadView(bytes.data + reader.position(), static_cast(payload_length)); + return input; +} + +class FunctionalParser { + public: + virtual ~FunctionalParser() = default; + + [[nodiscard]] virtual Status bind(const BindingInfo& info) = 0; + + [[nodiscard]] virtual Status parseObject(PayloadView, Timestamp, ObjectWriter&) { + return Status::error("object route is not implemented by this parser module"); + } + + [[nodiscard]] virtual Status parseScalars(PayloadView, Timestamp, ScalarWriter&) { + return Status::error("scalar route is not implemented by this parser module"); + } +}; + +namespace detail { + +template +class ModuleExports { + public: + struct Instance { + explicit Instance(uint32_t index) : claim_index(index) {} + + Parser parser; + uint32_t claim_index = 0; + Route route = Route::kScalar; + bool bound = false; + Blob output; + std::array error{}; + }; + + struct Slot { + Instance* instance = nullptr; + uint32_t generation = 1; + }; + + class TableLock { + public: + void lock() noexcept { +#if !defined(__wasm__) + while (flag_.test_and_set(std::memory_order_acquire)) {} +#endif + } + void unlock() noexcept { +#if !defined(__wasm__) + flag_.clear(std::memory_order_release); +#endif + } + + private: +#if !defined(__wasm__) + std::atomic_flag flag_ = ATOMIC_FLAG_INIT; +#endif + }; + + class LockGuard { + public: + explicit LockGuard(TableLock& lock) noexcept : lock_(lock) { + lock_.lock(); + } + ~LockGuard() { + lock_.unlock(); + } + + private: + TableLock& lock_; + }; + + struct Table { + ~Table() { + delete[] slots; + } + + [[nodiscard]] bool grow() noexcept { + const size_t next_capacity = capacity == 0 ? size_t{8} : capacity * 2; + if (next_capacity < capacity || next_capacity > UINT32_MAX) { + return false; + } + auto* allocation = new (std::nothrow) Slot[next_capacity]; + if (allocation == nullptr) { + return false; + } + for (size_t index = 0; index < size; ++index) { + allocation[index] = slots[index]; + } + delete[] slots; + slots = allocation; + capacity = next_capacity; + return true; + } + + Slot* slots = nullptr; + size_t size = 0; + size_t capacity = 0; + TableLock lock; + }; + + [[nodiscard]] static uint64_t create(uint32_t claim_index) noexcept { + if (claim_index >= static_cast(PJ_PARSER_MODULE_CLAIM_COUNT)) { + recordCreationError("claim index is outside the module manifest"); + return kCreationErrorToken; + } + PJ_PARSER_MODULE_TRY { + auto* instance = new (std::nothrow) Instance(claim_index); + if (instance == nullptr) { + recordCreationError("parser-module instance allocation failed"); + return kCreationErrorToken; + } + auto& table = instances(); + LockGuard guard(table.lock); + size_t slot_index = 0; + while (slot_index < table.size && + (table.slots[slot_index].instance != nullptr || table.slots[slot_index].generation == 0)) { + ++slot_index; + } + if (slot_index == table.size) { + if (table.size == table.capacity && !table.grow()) { + delete instance; + recordError(creationError(), "parser-module instance-table allocation failed"); + return kCreationErrorToken; + } + ++table.size; + } + auto& slot = table.slots[slot_index]; + slot.instance = instance; + if (slot_index >= UINT32_MAX) { + slot.instance = nullptr; + delete instance; + recordError(creationError(), "parser-module instance table is full"); + return kCreationErrorToken; + } + return tokenFor(slot_index, slot.generation); + } + PJ_PARSER_MODULE_CATCH_ALL { + recordCreationError("parser-module constructor failed"); + return kCreationErrorToken; + } + } + + static void destroy(uint64_t token) noexcept { + Instance* instance = nullptr; + { + auto& table = instances(); + LockGuard guard(table.lock); + Slot* slot = findSlotLocked(table, token); + if (slot == nullptr) { + recordError(badTokenError(), "pj_module_destroy received a stale or unknown instance token"); + return; + } + instance = slot->instance; + slot->instance = nullptr; + ++slot->generation; + // A wrapped generation retires the slot permanently so no stale token + // can become valid again. + } + PJ_PARSER_MODULE_TRY { + delete instance; + } + PJ_PARSER_MODULE_CATCH_ALL {} + } + + [[nodiscard]] static int32_t bind(uint64_t token, uint64_t address, uint64_t length) noexcept { + Instance* instance = find(token); + if (instance == nullptr) { + return failBadToken("pj_module_bind received a stale or unknown instance token"); + } + if (address == 0 || length > std::numeric_limits::max()) { + return fail(*instance, kModuleMalformedInput, "BindingInfo buffer is unreadable"); + } + PJ_PARSER_MODULE_TRY { + auto info = readBindingInfo( + ByteView(reinterpret_cast(static_cast(address)), static_cast(length))); + if (!info) { + return fail(*instance, kModuleMalformedInput, info.status().message()); + } + if (info->claimIndex() != instance->claim_index) { + return fail(*instance, kModuleBadClaimIndex, "BindingInfo claim index does not match the instance"); + } + const Status status = instance->parser.bind(*info); + instance->bound = status.isOk(); + instance->route = info->route(); + if (status.isOk()) { + instance->error[0] = '\0'; + return kModuleOk; + } + recordError(instance->error, status.message()); + return status.isDecline() ? kModuleDecline : kModuleError; + } + PJ_PARSER_MODULE_CATCH_ALL { + return fail(*instance, kModuleError, "parser-module bind threw an exception"); + } + } + + [[nodiscard]] static int32_t parse( + uint64_t token, uint64_t input_address, uint64_t input_length, uint64_t output_address_pointer, + uint64_t output_length_pointer) noexcept { + Instance* instance = find(token); + if (instance == nullptr) { + return failBadToken("pj_module_parse received a stale or unknown instance token"); + } + if (!instance->bound) { + return fail(*instance, kModuleError, "parser-module instance is not bound"); + } + if (input_address == 0 || output_address_pointer == 0 || output_length_pointer == 0 || + input_length > std::numeric_limits::max()) { + return fail(*instance, kModuleMalformedInput, "ParseInput buffer is unreadable"); + } + PJ_PARSER_MODULE_TRY { + auto input = readParseInput(ByteView( + reinterpret_cast(static_cast(input_address)), static_cast(input_length))); + if (!input) { + return fail(*instance, kModuleMalformedInput, input.status().message()); + } + + Expected output = Status::error("parser route is invalid"); + Status parsed = Status::error("parser route is invalid"); + if (instance->route == Route::kObject) { + ObjectWriter writer(input->payload); + parsed = + instance->parser.parseObject(input->payload, Timestamp{input->has_timestamp, input->timestamp_ns}, writer); + if (parsed.isOk()) { + output = writer.finish(); + } + } else { + ScalarWriter writer; + parsed = + instance->parser.parseScalars(input->payload, Timestamp{input->has_timestamp, input->timestamp_ns}, writer); + if (parsed.isOk()) { + output = writer.finish(); + } + } + if (!parsed.isOk()) { + return fail(*instance, kModuleError, parsed.message()); + } + if (!output) { + return fail(*instance, kModuleError, output.status().message()); + } + instance->output = std::move(*output); + const uint64_t output_address = addressOf(instance->output.data()); + const uint64_t output_length = instance->output.size(); + std::memcpy( + reinterpret_cast(static_cast(output_address_pointer)), &output_address, + sizeof(output_address)); + std::memcpy( + reinterpret_cast(static_cast(output_length_pointer)), &output_length, + sizeof(output_length)); + instance->error[0] = '\0'; + return kModuleOk; + } + PJ_PARSER_MODULE_CATCH_ALL { + return fail(*instance, kModuleError, "parser-module parse threw an exception"); + } + } + + [[nodiscard]] static uint64_t lastError(uint64_t token, uint64_t address, uint64_t capacity) noexcept { + if (address == 0 || capacity == 0 || capacity > std::numeric_limits::max()) { + return 0; + } + std::array source{}; + { + auto& table = instances(); + LockGuard guard(table.lock); + if (token == kCreationErrorToken) { + source = creationError(); + } else if (Slot* slot = findSlotLocked(table, token)) { + source = slot->instance->error; + } else { + source = badTokenError(); + } + } + size_t length = 0; + while (length < source.size() && source[length] != '\0') { + ++length; + } + const size_t written = length < static_cast(capacity) ? length : static_cast(capacity); + if (written != 0) { + std::memcpy(reinterpret_cast(static_cast(address)), source.data(), written); + } + return written; + } + + private: + [[nodiscard]] static Table& instances() { + static Table value; + return value; + } + + [[nodiscard]] static std::array& creationError() noexcept { + static std::array value{}; + return value; + } + + [[nodiscard]] static std::array& badTokenError() noexcept { + static std::array value{}; + return value; + } + + static void recordCreationError(std::string_view message) noexcept { + auto& table = instances(); + LockGuard guard(table.lock); + recordError(creationError(), message); + } + + [[nodiscard]] static uint64_t tokenFor(size_t index, uint32_t generation) noexcept { + return (static_cast(generation) << 32U) | static_cast(index + 1); + } + + [[nodiscard]] static Slot* findSlotLocked(Table& table, uint64_t token) noexcept { + const uint32_t encoded_index = static_cast(token); + const uint32_t generation = static_cast(token >> 32U); + if (encoded_index == 0 || generation == 0) { + return nullptr; + } + const size_t index = static_cast(encoded_index - 1); + if (index >= table.size) { + return nullptr; + } + Slot& slot = table.slots[index]; + return slot.instance != nullptr && slot.generation == generation ? &slot : nullptr; + } + + [[nodiscard]] static Instance* find(uint64_t token) noexcept { + // The table lock makes create/destroy safe alongside calls on different + // instances. Calls and lifecycle changes for the same token remain + // serialized by the host so the returned instance stays alive. + auto& table = instances(); + LockGuard guard(table.lock); + Slot* slot = findSlotLocked(table, token); + return slot == nullptr ? nullptr : slot->instance; + } + + [[nodiscard]] static int32_t failBadToken(std::string_view message) noexcept { + auto& table = instances(); + LockGuard guard(table.lock); + recordError(badTokenError(), message); + return kModuleBadToken; + } + + [[nodiscard]] static int32_t fail(Instance& instance, int32_t code, std::string_view message) noexcept { + recordError(instance.error, message); + return code; + } +}; + +} // namespace detail +} // namespace pj + +#if defined(__wasm__) +#define PJ_PARSER_MODULE_METADATA_EXPORTS +#else +#define PJ_PARSER_MODULE_METADATA_EXPORTS \ + PJ_PARSER_MODULE_EXPORT uint64_t pj_module_manifest_addr() noexcept { \ + return ::pj::detail::addressOf(::pj::detail::kBuiltManifest); \ + } \ + PJ_PARSER_MODULE_EXPORT uint64_t pj_module_manifest_len() noexcept { \ + return sizeof(::pj::detail::kBuiltManifest) - 1; \ + } +#endif + +#define PJ_FUNCTIONAL_PARSER(ParserClass) \ + extern "C" { \ + PJ_PARSER_MODULE_EXPORT uint32_t pj_module_abi() noexcept { \ + return ::pj::kModuleAbiVersion; \ + } \ + PJ_PARSER_MODULE_EXPORT uint64_t pj_module_create(uint32_t claim_index) noexcept { \ + return ::pj::detail::ModuleExports::create(claim_index); \ + } \ + PJ_PARSER_MODULE_EXPORT void pj_module_destroy(uint64_t instance) noexcept { \ + ::pj::detail::ModuleExports::destroy(instance); \ + } \ + PJ_PARSER_MODULE_EXPORT int32_t \ + pj_module_bind(uint64_t instance, uint64_t info_address, uint64_t info_length) noexcept { \ + return ::pj::detail::ModuleExports::bind(instance, info_address, info_length); \ + } \ + PJ_PARSER_MODULE_EXPORT int32_t pj_module_parse( \ + uint64_t instance, uint64_t input_address, uint64_t input_length, uint64_t output_address_pointer, \ + uint64_t output_length_pointer) noexcept { \ + return ::pj::detail::ModuleExports::parse( \ + instance, input_address, input_length, output_address_pointer, output_length_pointer); \ + } \ + PJ_PARSER_MODULE_EXPORT uint64_t \ + pj_module_last_error(uint64_t instance, uint64_t buffer_address, uint64_t buffer_capacity) noexcept { \ + return ::pj::detail::ModuleExports::lastError(instance, buffer_address, buffer_capacity); \ + } \ + PJ_PARSER_MODULE_EXPORT uint64_t pj_module_alloc(uint64_t size) noexcept { \ + if (size > static_cast(SIZE_MAX)) { \ + return 0; \ + } \ + auto* allocation = new (std::nothrow) uint8_t[static_cast(size)]; \ + return ::pj::detail::addressOf(allocation); \ + } \ + PJ_PARSER_MODULE_EXPORT void pj_module_free(uint64_t address, uint64_t) noexcept { \ + delete[] reinterpret_cast(static_cast(address)); \ + } \ + PJ_PARSER_MODULE_METADATA_EXPORTS \ + } diff --git a/pj_base/include/pj_base/parser_module/object_writer.hpp b/pj_base/include/pj_base/parser_module/object_writer.hpp new file mode 100644 index 00000000..36b24e41 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/object_writer.hpp @@ -0,0 +1,1595 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file object_writer.hpp + * @brief Fallible canonical-wire and module output-descriptor builders. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/core.hpp" + +namespace pj { + +class WireWriter { + public: + [[nodiscard]] const Status& status() const noexcept { + return status_; + } + + [[nodiscard]] Status tag(uint32_t field_number, uint8_t wire_type) { + if (field_number == 0 || field_number > UINT32_C(0x1FFFFFFF) || wire_type > 5 || wire_type == 3 || wire_type == 4) { + return fail("invalid canonical-wire tag"); + } + return varint((static_cast(field_number) << 3U) | wire_type); + } + + [[nodiscard]] Status varint(uint64_t value) { + if (!status_.isOk()) { + return status_; + } + do { + uint8_t byte = static_cast(value & UINT64_C(0x7F)); + value >>= 7U; + if (value != 0) { + byte |= UINT8_C(0x80); + } + Status pushed = bytes_.push(byte); + if (!pushed.isOk()) { + return fail(pushed.message()); + } + } while (value != 0); + return Status::ok(); + } + + [[nodiscard]] Status varintField(uint32_t field_number, uint64_t value) { + Status tagged = tag(field_number, 0); + return tagged.isOk() ? varint(value) : tagged; + } + + [[nodiscard]] Status fixed32Field(uint32_t field_number, uint32_t value) { + Status tagged = tag(field_number, 5); + if (!tagged.isOk()) { + return tagged; + } + return littleEndian(value, 4); + } + + [[nodiscard]] Status fixed64Field(uint32_t field_number, uint64_t value) { + Status tagged = tag(field_number, 1); + if (!tagged.isOk()) { + return tagged; + } + return littleEndian(value, 8); + } + + [[nodiscard]] Status rawFixed64(uint64_t value) { + return littleEndian(value, 8); + } + + [[nodiscard]] Status doubleField(uint32_t field_number, double value) { + uint64_t bits = 0; + std::memcpy(&bits, &value, sizeof(bits)); + return fixed64Field(field_number, bits); + } + + [[nodiscard]] Status floatField(uint32_t field_number, float value) { + uint32_t bits = 0; + std::memcpy(&bits, &value, sizeof(bits)); + return fixed32Field(field_number, bits); + } + + [[nodiscard]] Status lengthDelimited(uint32_t field_number, ByteView value) { + Status tagged = tag(field_number, 2); + if (!tagged.isOk()) { + return tagged; + } + Status length = varint(value.size); + if (!length.isOk()) { + return length; + } + Status appended = bytes_.append(value); + return appended.isOk() ? Status::ok() : fail(appended.message()); + } + + [[nodiscard]] Status stringField(uint32_t field_number, std::string_view value) { + return lengthDelimited(field_number, ByteView(reinterpret_cast(value.data()), value.size())); + } + + [[nodiscard]] Status messageField(uint32_t field_number, const WireWriter& nested) { + if (!nested.status().isOk()) { + return fail(nested.status().message()); + } + return lengthDelimited(field_number, nested.view()); + } + + [[nodiscard]] ByteView view() const noexcept { + return bytes_.view(); + } + + [[nodiscard]] Blob take() noexcept { + return std::move(bytes_); + } + + private: + [[nodiscard]] Status littleEndian(uint64_t value, size_t width) { + for (size_t index = 0; index < width; ++index) { + Status pushed = bytes_.push(static_cast(value >> (index * 8U))); + if (!pushed.isOk()) { + return fail(pushed.message()); + } + } + return Status::ok(); + } + + [[nodiscard]] Status fail(std::string_view message) { + if (status_.isOk()) { + status_ = Status::error(message); + } + return status_; + } + + Blob bytes_; + Status status_; +}; + +class ObjectWriter { + public: + static constexpr uint16_t kImageObjectType = 1; + static constexpr uint16_t kPointCloudObjectType = 3; + static constexpr uint16_t kDepthImageObjectType = 4; + static constexpr uint16_t kOccupancyGridObjectType = 7; + static constexpr uint16_t kCompressedPointCloudObjectType = 8; + static constexpr uint16_t kMesh3DObjectType = 9; + static constexpr uint16_t kVideoFrameObjectType = 10; + static constexpr uint16_t kOccupancyGridUpdateObjectType = 15; + static constexpr uint16_t kVoxelGridObjectType = 18; + static constexpr uint32_t kImageDataField = 7; + static constexpr uint32_t kPointCloudDataField = 9; + static constexpr uint32_t kDepthImageDataField = 5; + static constexpr uint32_t kOccupancyGridDataField = 7; + static constexpr uint32_t kCompressedPointCloudDataField = 4; + static constexpr uint32_t kMesh3DDataField = 7; + static constexpr uint32_t kVideoFrameDataField = 3; + static constexpr uint32_t kOccupancyGridUpdateDataField = 7; + static constexpr uint32_t kVoxelGridDataField = 12; + + enum class PointFieldDatatype : uint32_t { + kUnknown = 0, + kInt8 = 1, + kUint8 = 2, + kInt16 = 3, + kUint16 = 4, + kInt32 = 5, + kUint32 = 6, + kFloat32 = 7, + kFloat64 = 8, + }; + + class PointCloudBuilder; + class ImageBuilder; + class DepthImageBuilder; + class OccupancyGridBuilder; + class CompressedPointCloudBuilder; + class Mesh3DBuilder; + class VideoFrameBuilder; + class OccupancyGridUpdateBuilder; + class VoxelGridBuilder; + + explicit ObjectWriter(PayloadView input_payload = {}) : input_payload_(input_payload) {} + + [[nodiscard]] PointCloudBuilder pointCloud(); + [[nodiscard]] ImageBuilder image(); + [[nodiscard]] DepthImageBuilder depthImage(); + [[nodiscard]] OccupancyGridBuilder occupancyGrid(); + [[nodiscard]] CompressedPointCloudBuilder compressedPointCloud(); + [[nodiscard]] Mesh3DBuilder mesh3D(); + [[nodiscard]] VideoFrameBuilder videoFrame(); + [[nodiscard]] OccupancyGridUpdateBuilder occupancyGridUpdate(); + [[nodiscard]] VoxelGridBuilder voxelGrid(); + + [[nodiscard]] const Status& status() const noexcept { + return status_; + } + + /// Produce one object-route OutputDescriptor v1 block. + [[nodiscard]] Expected finish() { + if (!status_.isOk()) { + return status_; + } + if (kind_ == Kind::kNone) { + return Status::error("ObjectWriter has no selected object type"); + } + auto wire = writeSelected(); + if (!wire) { + return wire.status(); + } + Blob descriptor; + Status reserved = descriptor.reserve(40 + wire->size()); + if (!reserved.isOk()) { + return reserved; + } + const uint16_t object_type = selectedObjectType(); + const uint32_t splice_field = selectedSpliceField(); + const bool has_splice = splice_.has_value; + Status result = appendLittle(descriptor, 1, 2); + if (result.isOk()) { + result = descriptor.push(2); + } + if (result.isOk()) { + result = descriptor.push(0); + } + if (result.isOk()) { + result = appendLittle(descriptor, object_type, 2); + } + if (result.isOk()) { + result = appendLittle(descriptor, has_splice ? 1 : 0, 2); + } + if (result.isOk()) { + result = appendLittle(descriptor, has_splice ? splice_field : 0, 4); + } + if (result.isOk()) { + result = appendLittle(descriptor, has_splice ? splice_.reference.offset : 0, 8); + } + if (result.isOk()) { + result = appendLittle(descriptor, has_splice ? splice_.reference.length : 0, 8); + } + if (result.isOk()) { + result = appendLittle(descriptor, wire->size(), 8); + } + if (result.isOk()) { + result = descriptor.append(wire->view()); + } + return result.isOk() ? Expected(std::move(descriptor)) : Expected(result); + } + + private: + enum class Kind : uint8_t { + kNone, + kPointCloud, + kImage, + kDepthImage, + kOccupancyGrid, + kCompressedPointCloud, + kMesh3D, + kVideoFrame, + kOccupancyGridUpdate, + kVoxelGrid, + }; + + struct Vector3State { + double x = 0; + double y = 0; + double z = 0; + }; + + struct QuaternionState { + double x = 0; + double y = 0; + double z = 0; + double w = 1; + }; + + struct PoseState { + Vector3State position; + QuaternionState orientation; + }; + + struct ColorState { + double r = 0; + double g = 0; + double b = 0; + double a = 1; + }; + + struct PointFieldState { + std::string name; + uint32_t offset = 0; + PointFieldDatatype datatype = PointFieldDatatype::kUnknown; + uint32_t count = 1; + }; + + struct PointCloudState { + int64_t timestamp_ns = 0; + uint32_t width = 0; + uint32_t height = 1; + uint32_t point_step = 0; + uint32_t row_step = 0; + bool is_bigendian = false; + bool is_dense = true; + std::vector fields; + Blob data; + std::string frame_id; + }; + + struct ImageState { + int64_t timestamp_ns = 0; + uint32_t width = 0; + uint32_t height = 0; + std::string encoding; + uint32_t row_step = 0; + bool is_bigendian = false; + Blob data; + bool has_compressed_depth_min = false; + float compressed_depth_min = 0; + bool has_compressed_depth_max = false; + float compressed_depth_max = 0; + std::string frame_id; + }; + + struct DepthImageState { + int64_t timestamp_ns = 0; + uint32_t width = 0; + uint32_t height = 0; + std::string encoding; + Blob data; + std::array intrinsics{}; + std::string distortion_model; + std::vector distortion; + }; + + struct OccupancyGridState { + int64_t timestamp_ns = 0; + std::string frame_id; + PoseState origin; + double resolution = 0; + uint32_t width = 0; + uint32_t height = 0; + Blob data; + }; + + struct CompressedPointCloudState { + int64_t timestamp_ns = 0; + std::string frame_id; + std::string format; + Blob data; + }; + + struct Mesh3DState { + int64_t timestamp_ns = 0; + std::string frame_id; + std::string id; + PoseState pose; + Vector3State scale{1, 1, 1}; + std::string format; + Blob data; + std::string url; + ColorState color; + bool override_color = false; + }; + + struct VideoFrameState { + int64_t timestamp_ns = 0; + std::string frame_id; + Blob data; + std::string format; + }; + + struct OccupancyGridUpdateState { + int64_t timestamp_ns = 0; + std::string frame_id; + int32_t x = 0; + int32_t y = 0; + uint32_t width = 0; + uint32_t height = 0; + Blob data; + }; + + struct VoxelGridState { + int64_t timestamp_ns = 0; + std::string frame_id; + PoseState origin; + Vector3State cell_size; + uint32_t column_count = 0; + uint32_t row_count = 0; + uint32_t slice_count = 0; + uint32_t cell_stride = 0; + uint32_t row_stride = 0; + uint32_t slice_stride = 0; + std::vector fields; + Blob data; + }; + + struct SpliceState { + bool has_value = false; + InputSpanRef reference; + }; + + enum class DataSource : uint8_t { + kNone, + kCopied, + kSpliced, + }; + + [[nodiscard]] Status select(Kind kind) { + if (kind_ != Kind::kNone && kind_ != kind) { + return fail("ObjectWriter may emit only one object per parse call"); + } + kind_ = kind; + return status_; + } + + [[nodiscard]] Status fail(std::string_view message) { + if (status_.isOk()) { + status_ = Status::error(message); + } + return status_; + } + + [[nodiscard]] Status setString(std::string& destination, std::string_view value) { + if (!status_.isOk()) { + return status_; + } + PJ_PARSER_MODULE_TRY { + destination.assign(value.data(), value.size()); + return Status::ok(); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return fail("ObjectWriter string allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return fail("ObjectWriter string assignment failed"); + } + } + + [[nodiscard]] Status setData(Blob& destination, PayloadView value) { + if (data_source_ != DataSource::kNone) { + return fail("pj.parser.contract_violation: object bulk data source was selected more than once"); + } + auto allocated = allocateBlob(value.size); + if (!allocated) { + return fail(allocated.status().message()); + } + if (value.size != 0) { + if (value.data == nullptr) { + return fail("ObjectWriter data source is null"); + } + std::memcpy(allocated->data(), value.data, value.size); + } + destination = std::move(*allocated); + data_source_ = DataSource::kCopied; + return Status::ok(); + } + + [[nodiscard]] Status setDataFromInput(InputSpanRef reference) { + if (data_source_ != DataSource::kNone) { + return fail("pj.parser.contract_violation: object bulk data source was selected more than once"); + } + if (reference.offset > input_payload_.size || reference.length > input_payload_.size - reference.offset) { + return fail("pj.parser.contract_violation: object splice range is outside the parse payload"); + } + if (input_payload_.data == nullptr && input_payload_.size != 0) { + return fail("pj.parser.contract_violation: object splice payload storage is null"); + } + splice_.has_value = true; + splice_.reference = reference; + data_source_ = DataSource::kSpliced; + return Status::ok(); + } + + [[nodiscard]] static Status appendLittle(Blob& output, uint64_t value, size_t width) { + for (size_t index = 0; index < width; ++index) { + Status status = output.push(static_cast(value >> (index * 8U))); + if (!status.isOk()) { + return status; + } + } + return Status::ok(); + } + + [[nodiscard]] static Status writeTimestamp(WireWriter& writer, int64_t timestamp_ns) { + constexpr int64_t kNanosPerSecond = INT64_C(1000000000); + int64_t seconds = timestamp_ns / kNanosPerSecond; + int32_t nanos = static_cast(timestamp_ns % kNanosPerSecond); + if (nanos < 0) { + --seconds; + nanos += static_cast(kNanosPerSecond); + } + WireWriter nested; + Status status = nested.varintField(1, static_cast(seconds)); + if (status.isOk()) { + status = nested.varintField(2, static_cast(nanos)); + } + return status.isOk() ? writer.messageField(1, nested) : status; + } + + [[nodiscard]] static Status writeVector3(WireWriter& writer, uint32_t field, const Vector3State& value) { + WireWriter nested; + Status status = nested.doubleField(1, value.x); + if (status.isOk()) { + status = nested.doubleField(2, value.y); + } + if (status.isOk()) { + status = nested.doubleField(3, value.z); + } + return status.isOk() ? writer.messageField(field, nested) : status; + } + + [[nodiscard]] static Status writePose(WireWriter& writer, uint32_t field, const PoseState& value) { + WireWriter nested; + Status status = writeVector3(nested, 1, value.position); + if (status.isOk()) { + WireWriter quaternion; + status = quaternion.doubleField(1, value.orientation.x); + if (status.isOk()) { + status = quaternion.doubleField(2, value.orientation.y); + } + if (status.isOk()) { + status = quaternion.doubleField(3, value.orientation.z); + } + if (status.isOk()) { + status = quaternion.doubleField(4, value.orientation.w); + } + if (status.isOk()) { + status = nested.messageField(2, quaternion); + } + } + return status.isOk() ? writer.messageField(field, nested) : status; + } + + [[nodiscard]] static Status writeColor(WireWriter& writer, uint32_t field, const ColorState& value) { + WireWriter nested; + Status status = nested.doubleField(1, value.r); + if (status.isOk()) { + status = nested.doubleField(2, value.g); + } + if (status.isOk()) { + status = nested.doubleField(3, value.b); + } + if (status.isOk()) { + status = nested.doubleField(4, value.a); + } + return status.isOk() ? writer.messageField(field, nested) : status; + } + + [[nodiscard]] static Status writePointField(WireWriter& writer, uint32_t field_number, const PointFieldState& field) { + WireWriter nested; + Status status = nested.stringField(1, field.name); + if (status.isOk()) { + status = nested.varintField(2, field.offset); + } + if (status.isOk()) { + status = nested.varintField(3, static_cast(field.datatype)); + } + if (status.isOk()) { + status = nested.varintField(4, field.count); + } + return status.isOk() ? writer.messageField(field_number, nested) : status; + } + + [[nodiscard]] static Status writePackedDoubles( + WireWriter& writer, uint32_t field, const double* values, size_t count) { + WireWriter packed; + Status status; + for (size_t index = 0; status.isOk() && index < count; ++index) { + uint64_t bits = 0; + std::memcpy(&bits, values + index, sizeof(bits)); + status = packed.rawFixed64(bits); + } + return status.isOk() ? writer.lengthDelimited(field, packed.view()) : status; + } + + [[nodiscard]] Expected writeSelected() { + switch (kind_) { + case Kind::kPointCloud: + return writePointCloud(); + case Kind::kImage: + return writeImage(); + case Kind::kDepthImage: + return writeDepthImage(); + case Kind::kOccupancyGrid: + return writeOccupancyGrid(); + case Kind::kCompressedPointCloud: + return writeCompressedPointCloud(); + case Kind::kMesh3D: + return writeMesh3D(); + case Kind::kVideoFrame: + return writeVideoFrame(); + case Kind::kOccupancyGridUpdate: + return writeOccupancyGridUpdate(); + case Kind::kVoxelGrid: + return writeVoxelGrid(); + case Kind::kNone: + break; + } + return Status::error("ObjectWriter has no selected object type"); + } + + [[nodiscard]] uint16_t selectedObjectType() const noexcept { + switch (kind_) { + case Kind::kImage: + return kImageObjectType; + case Kind::kPointCloud: + return kPointCloudObjectType; + case Kind::kDepthImage: + return kDepthImageObjectType; + case Kind::kOccupancyGrid: + return kOccupancyGridObjectType; + case Kind::kCompressedPointCloud: + return kCompressedPointCloudObjectType; + case Kind::kMesh3D: + return kMesh3DObjectType; + case Kind::kVideoFrame: + return kVideoFrameObjectType; + case Kind::kOccupancyGridUpdate: + return kOccupancyGridUpdateObjectType; + case Kind::kVoxelGrid: + return kVoxelGridObjectType; + case Kind::kNone: + return 0; + } + return 0; + } + + [[nodiscard]] uint32_t selectedSpliceField() const noexcept { + switch (kind_) { + case Kind::kImage: + return kImageDataField; + case Kind::kPointCloud: + return kPointCloudDataField; + case Kind::kDepthImage: + return kDepthImageDataField; + case Kind::kOccupancyGrid: + return kOccupancyGridDataField; + case Kind::kCompressedPointCloud: + return kCompressedPointCloudDataField; + case Kind::kMesh3D: + return kMesh3DDataField; + case Kind::kVideoFrame: + return kVideoFrameDataField; + case Kind::kOccupancyGridUpdate: + return kOccupancyGridUpdateDataField; + case Kind::kVoxelGrid: + return kVoxelGridDataField; + case Kind::kNone: + return 0; + } + return 0; + } + + [[nodiscard]] Expected writePointCloud() { + WireWriter writer; + Status status = writeTimestamp(writer, point_cloud_.timestamp_ns); + if (status.isOk()) { + status = writer.varintField(2, point_cloud_.width); + } + if (status.isOk()) { + status = writer.varintField(3, point_cloud_.height); + } + if (status.isOk()) { + status = writer.varintField(4, point_cloud_.point_step); + } + if (status.isOk()) { + status = writer.varintField(5, point_cloud_.row_step); + } + if (status.isOk()) { + status = writer.varintField(6, point_cloud_.is_bigendian ? 1 : 0); + } + if (status.isOk()) { + status = writer.varintField(7, point_cloud_.is_dense ? 1 : 0); + } + for (const auto& field : point_cloud_.fields) { + if (!status.isOk()) { + break; + } + WireWriter nested; + status = nested.stringField(1, field.name); + if (status.isOk()) { + status = nested.varintField(2, field.offset); + } + if (status.isOk()) { + status = nested.varintField(3, static_cast(field.datatype)); + } + if (status.isOk()) { + status = nested.varintField(4, field.count); + } + if (status.isOk()) { + status = writer.messageField(8, nested); + } + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(9, point_cloud_.data.view()); + } + if (status.isOk()) { + status = writer.stringField(10, point_cloud_.frame_id); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeImage() { + WireWriter writer; + Status status = writeTimestamp(writer, image_.timestamp_ns); + if (status.isOk()) { + status = writer.varintField(2, image_.width); + } + if (status.isOk()) { + status = writer.varintField(3, image_.height); + } + if (status.isOk()) { + status = writer.stringField(4, image_.encoding); + } + if (status.isOk()) { + status = writer.varintField(5, image_.row_step); + } + if (status.isOk()) { + status = writer.varintField(6, image_.is_bigendian ? 1 : 0); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(7, image_.data.view()); + } + if (status.isOk() && image_.has_compressed_depth_min) { + uint32_t bits = 0; + std::memcpy(&bits, &image_.compressed_depth_min, sizeof(bits)); + status = writer.fixed32Field(8, bits); + } + if (status.isOk() && image_.has_compressed_depth_max) { + uint32_t bits = 0; + std::memcpy(&bits, &image_.compressed_depth_max, sizeof(bits)); + status = writer.fixed32Field(9, bits); + } + if (status.isOk()) { + status = writer.stringField(10, image_.frame_id); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeDepthImage() { + WireWriter writer; + Status status = writeTimestamp(writer, depth_image_.timestamp_ns); + if (status.isOk()) { + status = writer.varintField(2, depth_image_.width); + } + if (status.isOk()) { + status = writer.varintField(3, depth_image_.height); + } + if (status.isOk()) { + status = writer.stringField(4, depth_image_.encoding); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(5, depth_image_.data.view()); + } + if (status.isOk()) { + status = writePackedDoubles(writer, 6, depth_image_.intrinsics.data(), depth_image_.intrinsics.size()); + } + if (status.isOk()) { + status = writer.stringField(7, depth_image_.distortion_model); + } + if (status.isOk()) { + status = writePackedDoubles(writer, 8, depth_image_.distortion.data(), depth_image_.distortion.size()); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeOccupancyGrid() { + WireWriter writer; + Status status = writeTimestamp(writer, occupancy_grid_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, occupancy_grid_.frame_id); + } + if (status.isOk()) { + status = writePose(writer, 3, occupancy_grid_.origin); + } + if (status.isOk()) { + status = writer.doubleField(4, occupancy_grid_.resolution); + } + if (status.isOk()) { + status = writer.varintField(5, occupancy_grid_.width); + } + if (status.isOk()) { + status = writer.varintField(6, occupancy_grid_.height); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(7, occupancy_grid_.data.view()); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeCompressedPointCloud() { + WireWriter writer; + Status status = writeTimestamp(writer, compressed_point_cloud_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, compressed_point_cloud_.frame_id); + } + if (status.isOk()) { + status = writer.stringField(3, compressed_point_cloud_.format); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(4, compressed_point_cloud_.data.view()); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeMesh3D() { + WireWriter writer; + Status status = writeTimestamp(writer, mesh3d_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, mesh3d_.frame_id); + } + if (status.isOk()) { + status = writer.stringField(3, mesh3d_.id); + } + if (status.isOk()) { + status = writePose(writer, 4, mesh3d_.pose); + } + if (status.isOk()) { + status = writeVector3(writer, 5, mesh3d_.scale); + } + if (status.isOk()) { + status = writer.stringField(6, mesh3d_.format); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(7, mesh3d_.data.view()); + } + if (status.isOk()) { + status = writer.stringField(8, mesh3d_.url); + } + if (status.isOk()) { + status = writeColor(writer, 9, mesh3d_.color); + } + if (status.isOk()) { + status = writer.varintField(10, mesh3d_.override_color ? 1 : 0); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeVideoFrame() { + WireWriter writer; + Status status = writeTimestamp(writer, video_frame_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, video_frame_.frame_id); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(3, video_frame_.data.view()); + } + if (status.isOk()) { + status = writer.stringField(4, video_frame_.format); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeOccupancyGridUpdate() { + WireWriter writer; + Status status = writeTimestamp(writer, occupancy_grid_update_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, occupancy_grid_update_.frame_id); + } + if (status.isOk()) { + status = writer.varintField(3, static_cast(occupancy_grid_update_.x)); + } + if (status.isOk()) { + status = writer.varintField(4, static_cast(occupancy_grid_update_.y)); + } + if (status.isOk()) { + status = writer.varintField(5, occupancy_grid_update_.width); + } + if (status.isOk()) { + status = writer.varintField(6, occupancy_grid_update_.height); + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(7, occupancy_grid_update_.data.view()); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + [[nodiscard]] Expected writeVoxelGrid() { + WireWriter writer; + Status status = writeTimestamp(writer, voxel_grid_.timestamp_ns); + if (status.isOk()) { + status = writer.stringField(2, voxel_grid_.frame_id); + } + if (status.isOk()) { + status = writePose(writer, 3, voxel_grid_.origin); + } + if (status.isOk()) { + status = writeVector3(writer, 4, voxel_grid_.cell_size); + } + if (status.isOk()) { + status = writer.varintField(5, voxel_grid_.column_count); + } + if (status.isOk()) { + status = writer.varintField(6, voxel_grid_.row_count); + } + if (status.isOk()) { + status = writer.varintField(7, voxel_grid_.slice_count); + } + if (status.isOk()) { + status = writer.varintField(8, voxel_grid_.cell_stride); + } + if (status.isOk()) { + status = writer.varintField(9, voxel_grid_.row_stride); + } + if (status.isOk()) { + status = writer.varintField(10, voxel_grid_.slice_stride); + } + for (const auto& field : voxel_grid_.fields) { + if (status.isOk()) { + status = writePointField(writer, 11, field); + } + } + if (status.isOk() && !splice_.has_value) { + status = writer.lengthDelimited(12, voxel_grid_.data.view()); + } + return status.isOk() ? Expected(writer.take()) : Expected(status); + } + + Kind kind_ = Kind::kNone; + Status status_; + PayloadView input_payload_; + DataSource data_source_ = DataSource::kNone; + PointCloudState point_cloud_; + ImageState image_; + DepthImageState depth_image_; + OccupancyGridState occupancy_grid_; + CompressedPointCloudState compressed_point_cloud_; + Mesh3DState mesh3d_; + VideoFrameState video_frame_; + OccupancyGridUpdateState occupancy_grid_update_; + VoxelGridState voxel_grid_; + SpliceState splice_; + + friend class PointCloudBuilder; + friend class ImageBuilder; + friend class DepthImageBuilder; + friend class OccupancyGridBuilder; + friend class CompressedPointCloudBuilder; + friend class Mesh3DBuilder; + friend class VideoFrameBuilder; + friend class OccupancyGridUpdateBuilder; + friend class VoxelGridBuilder; + friend class ScalarWriter; +}; + +class ObjectWriter::PointCloudBuilder { + public: + explicit PointCloudBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kPointCloud); + } + + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->point_cloud_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setWidth(uint32_t value) { + owner_->point_cloud_.width = value; + return owner_->status_; + } + [[nodiscard]] Status setHeight(uint32_t value) { + owner_->point_cloud_.height = value; + return owner_->status_; + } + [[nodiscard]] Status setPointStep(uint32_t value) { + owner_->point_cloud_.point_step = value; + return owner_->status_; + } + [[nodiscard]] Status setRowStep(uint32_t value) { + owner_->point_cloud_.row_step = value; + return owner_->status_; + } + [[nodiscard]] Status setBigEndian(bool value) { + owner_->point_cloud_.is_bigendian = value; + return owner_->status_; + } + [[nodiscard]] Status setDense(bool value) { + owner_->point_cloud_.is_dense = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->point_cloud_.frame_id, value); + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->point_cloud_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef reference) { + return owner_->setDataFromInput(reference); + } + [[nodiscard]] Status addField( + std::string_view name, uint32_t offset, PointFieldDatatype datatype, uint32_t count = 1) { + if (!owner_->status_.isOk()) { + return owner_->status_; + } + PJ_PARSER_MODULE_TRY { + PointFieldState field; + field.name.assign(name.data(), name.size()); + field.offset = offset; + field.datatype = datatype; + field.count = count; + owner_->point_cloud_.fields.push_back(std::move(field)); + return Status::ok(); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return owner_->fail("ObjectWriter point-field allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return owner_->fail("ObjectWriter point-field creation failed"); + } + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::ImageBuilder { + public: + explicit ImageBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kImage); + } + + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->image_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setWidth(uint32_t value) { + owner_->image_.width = value; + return owner_->status_; + } + [[nodiscard]] Status setHeight(uint32_t value) { + owner_->image_.height = value; + return owner_->status_; + } + [[nodiscard]] Status setEncoding(std::string_view value) { + return owner_->setString(owner_->image_.encoding, value); + } + [[nodiscard]] Status setRowStep(uint32_t value) { + owner_->image_.row_step = value; + return owner_->status_; + } + [[nodiscard]] Status setBigEndian(bool value) { + owner_->image_.is_bigendian = value; + return owner_->status_; + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->image_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef reference) { + return owner_->setDataFromInput(reference); + } + [[nodiscard]] Status setCompressedDepthMin(float value) { + owner_->image_.has_compressed_depth_min = true; + owner_->image_.compressed_depth_min = value; + return owner_->status_; + } + [[nodiscard]] Status setCompressedDepthMax(float value) { + owner_->image_.has_compressed_depth_max = true; + owner_->image_.compressed_depth_max = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->image_.frame_id, value); + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::DepthImageBuilder { + public: + explicit DepthImageBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kDepthImage); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->depth_image_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setWidth(uint32_t value) { + owner_->depth_image_.width = value; + return owner_->status_; + } + [[nodiscard]] Status setHeight(uint32_t value) { + owner_->depth_image_.height = value; + return owner_->status_; + } + [[nodiscard]] Status setEncoding(std::string_view value) { + return owner_->setString(owner_->depth_image_.encoding, value); + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->depth_image_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + [[nodiscard]] Status setIntrinsics(const std::array& value) { + owner_->depth_image_.intrinsics = value; + return owner_->status_; + } + [[nodiscard]] Status setDistortionModel(std::string_view value) { + return owner_->setString(owner_->depth_image_.distortion_model, value); + } + [[nodiscard]] Status addDistortionCoefficient(double value) { + if (!owner_->status_.isOk()) { + return owner_->status_; + } + PJ_PARSER_MODULE_TRY { + owner_->depth_image_.distortion.push_back(value); + return Status::ok(); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return owner_->fail("ObjectWriter distortion allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return owner_->fail("ObjectWriter distortion creation failed"); + } + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::OccupancyGridBuilder { + public: + explicit OccupancyGridBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kOccupancyGrid); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->occupancy_grid_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->occupancy_grid_.frame_id, value); + } + [[nodiscard]] Status setOrigin( + double px, double py, double pz, double qx = 0, double qy = 0, double qz = 0, double qw = 1) { + owner_->occupancy_grid_.origin = {{px, py, pz}, {qx, qy, qz, qw}}; + return owner_->status_; + } + [[nodiscard]] Status setResolution(double value) { + owner_->occupancy_grid_.resolution = value; + return owner_->status_; + } + [[nodiscard]] Status setWidth(uint32_t value) { + owner_->occupancy_grid_.width = value; + return owner_->status_; + } + [[nodiscard]] Status setHeight(uint32_t value) { + owner_->occupancy_grid_.height = value; + return owner_->status_; + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->occupancy_grid_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::CompressedPointCloudBuilder { + public: + explicit CompressedPointCloudBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kCompressedPointCloud); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->compressed_point_cloud_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->compressed_point_cloud_.frame_id, value); + } + [[nodiscard]] Status setFormat(std::string_view value) { + return owner_->setString(owner_->compressed_point_cloud_.format, value); + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->compressed_point_cloud_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::Mesh3DBuilder { + public: + explicit Mesh3DBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kMesh3D); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->mesh3d_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->mesh3d_.frame_id, value); + } + [[nodiscard]] Status setId(std::string_view value) { + return owner_->setString(owner_->mesh3d_.id, value); + } + [[nodiscard]] Status setPose( + double px, double py, double pz, double qx = 0, double qy = 0, double qz = 0, double qw = 1) { + owner_->mesh3d_.pose = {{px, py, pz}, {qx, qy, qz, qw}}; + return owner_->status_; + } + [[nodiscard]] Status setScale(double x, double y, double z) { + owner_->mesh3d_.scale = {x, y, z}; + return owner_->status_; + } + [[nodiscard]] Status setFormat(std::string_view value) { + return owner_->setString(owner_->mesh3d_.format, value); + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->mesh3d_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + [[nodiscard]] Status setUrl(std::string_view value) { + return owner_->setString(owner_->mesh3d_.url, value); + } + [[nodiscard]] Status setColor(double r, double g, double b, double a) { + owner_->mesh3d_.color = {r, g, b, a}; + return owner_->status_; + } + [[nodiscard]] Status setOverrideColor(bool value) { + owner_->mesh3d_.override_color = value; + return owner_->status_; + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::VideoFrameBuilder { + public: + explicit VideoFrameBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kVideoFrame); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->video_frame_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->video_frame_.frame_id, value); + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->video_frame_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + [[nodiscard]] Status setFormat(std::string_view value) { + return owner_->setString(owner_->video_frame_.format, value); + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::OccupancyGridUpdateBuilder { + public: + explicit OccupancyGridUpdateBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kOccupancyGridUpdate); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->occupancy_grid_update_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->occupancy_grid_update_.frame_id, value); + } + [[nodiscard]] Status setX(int32_t value) { + owner_->occupancy_grid_update_.x = value; + return owner_->status_; + } + [[nodiscard]] Status setY(int32_t value) { + owner_->occupancy_grid_update_.y = value; + return owner_->status_; + } + [[nodiscard]] Status setWidth(uint32_t value) { + owner_->occupancy_grid_update_.width = value; + return owner_->status_; + } + [[nodiscard]] Status setHeight(uint32_t value) { + owner_->occupancy_grid_update_.height = value; + return owner_->status_; + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->occupancy_grid_update_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + + private: + ObjectWriter* owner_; +}; + +class ObjectWriter::VoxelGridBuilder { + public: + explicit VoxelGridBuilder(ObjectWriter& owner) : owner_(&owner) { + (void)owner_->select(Kind::kVoxelGrid); + } + [[nodiscard]] Status setTimestamp(int64_t value) { + owner_->voxel_grid_.timestamp_ns = value; + return owner_->status_; + } + [[nodiscard]] Status setFrameId(std::string_view value) { + return owner_->setString(owner_->voxel_grid_.frame_id, value); + } + [[nodiscard]] Status setOrigin( + double px, double py, double pz, double qx = 0, double qy = 0, double qz = 0, double qw = 1) { + owner_->voxel_grid_.origin = {{px, py, pz}, {qx, qy, qz, qw}}; + return owner_->status_; + } + [[nodiscard]] Status setCellSize(double x, double y, double z) { + owner_->voxel_grid_.cell_size = {x, y, z}; + return owner_->status_; + } + [[nodiscard]] Status setColumnCount(uint32_t value) { + owner_->voxel_grid_.column_count = value; + return owner_->status_; + } + [[nodiscard]] Status setRowCount(uint32_t value) { + owner_->voxel_grid_.row_count = value; + return owner_->status_; + } + [[nodiscard]] Status setSliceCount(uint32_t value) { + owner_->voxel_grid_.slice_count = value; + return owner_->status_; + } + [[nodiscard]] Status setCellStride(uint32_t value) { + owner_->voxel_grid_.cell_stride = value; + return owner_->status_; + } + [[nodiscard]] Status setRowStride(uint32_t value) { + owner_->voxel_grid_.row_stride = value; + return owner_->status_; + } + [[nodiscard]] Status setSliceStride(uint32_t value) { + owner_->voxel_grid_.slice_stride = value; + return owner_->status_; + } + [[nodiscard]] Status addField( + std::string_view name, uint32_t offset, PointFieldDatatype datatype, uint32_t count = 1) { + if (!owner_->status_.isOk()) { + return owner_->status_; + } + PJ_PARSER_MODULE_TRY { + PointFieldState field; + field.name.assign(name.data(), name.size()); + field.offset = offset; + field.datatype = datatype; + field.count = count; + owner_->voxel_grid_.fields.push_back(std::move(field)); + return Status::ok(); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return owner_->fail("ObjectWriter voxel-field allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return owner_->fail("ObjectWriter voxel-field creation failed"); + } + } + [[nodiscard]] Status setData(PayloadView value) { + return owner_->setData(owner_->voxel_grid_.data, value); + } + [[nodiscard]] Status setDataFromInput(InputSpanRef value) { + return owner_->setDataFromInput(value); + } + + private: + ObjectWriter* owner_; +}; + +inline ObjectWriter::PointCloudBuilder ObjectWriter::pointCloud() { + return PointCloudBuilder(*this); +} + +inline ObjectWriter::ImageBuilder ObjectWriter::image() { + return ImageBuilder(*this); +} + +inline ObjectWriter::DepthImageBuilder ObjectWriter::depthImage() { + return DepthImageBuilder(*this); +} + +inline ObjectWriter::OccupancyGridBuilder ObjectWriter::occupancyGrid() { + return OccupancyGridBuilder(*this); +} + +inline ObjectWriter::CompressedPointCloudBuilder ObjectWriter::compressedPointCloud() { + return CompressedPointCloudBuilder(*this); +} + +inline ObjectWriter::Mesh3DBuilder ObjectWriter::mesh3D() { + return Mesh3DBuilder(*this); +} + +inline ObjectWriter::VideoFrameBuilder ObjectWriter::videoFrame() { + return VideoFrameBuilder(*this); +} + +inline ObjectWriter::OccupancyGridUpdateBuilder ObjectWriter::occupancyGridUpdate() { + return OccupancyGridUpdateBuilder(*this); +} + +inline ObjectWriter::VoxelGridBuilder ObjectWriter::voxelGrid() { + return VoxelGridBuilder(*this); +} + +class ScalarWriter { + public: + [[nodiscard]] Status setTimestamp(int64_t timestamp_ns) { + has_timestamp_ = true; + timestamp_ns_ = timestamp_ns; + return status_; + } + + [[nodiscard]] Status add(std::string_view name, double value) { + return addValue(name, value); + } + [[nodiscard]] Status add(std::string_view name, int64_t value) { + return addValue(name, value); + } + [[nodiscard]] Status add(std::string_view name, uint64_t value) { + return addValue(name, value); + } + [[nodiscard]] Status add(std::string_view name, bool value) { + return addValue(name, value); + } + [[nodiscard]] Status add(std::string_view name, const char* value) { + return add(name, std::string_view(value == nullptr ? "" : value)); + } + [[nodiscard]] Status add(std::string_view name, std::string_view value) { + PJ_PARSER_MODULE_TRY { + return addValue(name, std::string(value)); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return fail("ScalarWriter string allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return fail("ScalarWriter string creation failed"); + } + } + + [[nodiscard]] Expected finish() { + if (!status_.isOk()) { + return status_; + } + size_t names_offset = 24; + for (const auto& field : fields_) { + size_t value_size = 8; + if (std::holds_alternative(field.value)) { + value_size = 1; + } else if (const auto* string = std::get_if(&field.value)) { + if (string->size() > std::numeric_limits::max()) { + return Status::error("scalar string exceeds the descriptor length range"); + } + value_size = 4 + string->size(); + } + if (names_offset > std::numeric_limits::max() - 9 - value_size) { + return Status::error("scalar output descriptor size overflow"); + } + names_offset += 9 + value_size; + } + size_t total_size = names_offset; + for (const auto& field : fields_) { + if (field.name.size() > std::numeric_limits::max() || + field.name.size() > std::numeric_limits::max() - total_size) { + return Status::error("scalar field names exceed the descriptor offset range"); + } + total_size += field.name.size(); + } + if (total_size > std::numeric_limits::max() || fields_.size() > std::numeric_limits::max()) { + return Status::error("scalar output descriptor exceeds the v1 offset range"); + } + + Blob output; + Status status = output.reserve(total_size); + if (status.isOk()) { + status = ObjectWriter::appendLittle(output, 1, 2); + } + if (status.isOk()) { + status = output.push(1); + } + if (status.isOk()) { + status = output.push(0); + } + if (status.isOk()) { + status = output.push(has_timestamp_ ? 1 : 0); + } + for (size_t index = 0; status.isOk() && index < 7; ++index) { + status = output.push(0); + } + if (status.isOk()) { + status = ObjectWriter::appendLittle(output, static_cast(timestamp_ns_), 8); + } + if (status.isOk()) { + status = ObjectWriter::appendLittle(output, fields_.size(), 4); + } + size_t current_name = names_offset; + for (const auto& field : fields_) { + if (!status.isOk()) { + break; + } + status = ObjectWriter::appendLittle(output, current_name, 4); + if (status.isOk()) { + status = ObjectWriter::appendLittle(output, field.name.size(), 4); + } + if (const auto* floating_value = std::get_if(&field.value)) { + status = status.isOk() ? output.push(0) : status; + uint64_t bits = 0; + std::memcpy(&bits, floating_value, sizeof(bits)); + status = status.isOk() ? ObjectWriter::appendLittle(output, bits, 8) : status; + } else if (const auto* signed_value = std::get_if(&field.value)) { + status = status.isOk() ? output.push(1) : status; + status = status.isOk() ? ObjectWriter::appendLittle(output, static_cast(*signed_value), 8) : status; + } else if (const auto* unsigned_value = std::get_if(&field.value)) { + status = status.isOk() ? output.push(2) : status; + status = status.isOk() ? ObjectWriter::appendLittle(output, *unsigned_value, 8) : status; + } else if (const auto* bool_value = std::get_if(&field.value)) { + status = status.isOk() ? output.push(3) : status; + status = status.isOk() ? output.push(*bool_value ? 1 : 0) : status; + } else { + const auto& string_value = std::get(field.value); + status = status.isOk() ? output.push(4) : status; + status = status.isOk() ? ObjectWriter::appendLittle(output, string_value.size(), 4) : status; + status = + status.isOk() + ? output.append(ByteView(reinterpret_cast(string_value.data()), string_value.size())) + : status; + } + current_name += field.name.size(); + } + for (const auto& field : fields_) { + status = status.isOk() + ? output.append(ByteView(reinterpret_cast(field.name.data()), field.name.size())) + : status; + } + return status.isOk() ? Expected(std::move(output)) : Expected(status); + } + + private: + using Value = std::variant; + struct Field { + std::string name; + Value value; + }; + + template + [[nodiscard]] Status addValue(std::string_view name, ValueType value) { + if (!status_.isOk()) { + return status_; + } + PJ_PARSER_MODULE_TRY { + Field field; + field.name.assign(name.data(), name.size()); + field.value = std::move(value); + fields_.push_back(std::move(field)); + return Status::ok(); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return fail("ScalarWriter allocation failed"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return fail("ScalarWriter field creation failed"); + } + } + + [[nodiscard]] Status fail(std::string_view message) { + if (status_.isOk()) { + status_ = Status::error(message); + } + return status_; + } + + bool has_timestamp_ = false; + int64_t timestamp_ns_ = 0; + std::vector fields_; + Status status_; + + friend class ObjectWriter; +}; + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/proto_field_locator.hpp b/pj_base/include/pj_base/parser_module/proto_field_locator.hpp new file mode 100644 index 00000000..9cb248f0 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/proto_field_locator.hpp @@ -0,0 +1,410 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file proto_field_locator.hpp + * @brief FileDescriptorSet field-path compiler for protobuf messages. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/proto_reader.hpp" + +namespace pj { + +using ProtoFieldId = size_t; + +namespace detail { + +struct ProtoSchemaField { + std::string name; + uint32_t number = 0; + uint32_t type = 0; + std::string type_name; + bool repeated = false; +}; + +struct ProtoSchemaMessage { + std::string full_name; + std::vector fields; + bool top_level = false; +}; + +struct ProtoRequestedField { + std::string path; + std::vector numbers; +}; + +inline std::string protoString(ByteView bytes) { + return std::string(reinterpret_cast(bytes.data), bytes.size); +} + +inline std::string normalizeProtoTypeName(std::string_view name) { + if (!name.empty() && name.front() == '.') { + name.remove_prefix(1); + } + return std::string(name); +} + +} // namespace detail + +class ProtoTraversalPlan { + public: + [[nodiscard]] size_t size() const noexcept { + return requested_.size(); + } + + [[nodiscard]] Expected field(std::string_view path) const { + for (size_t index = 0; index < requested_.size(); ++index) { + if (requested_[index].path == path) { + return index; + } + } + return Status::error("field path is not present in the protobuf traversal plan"); + } + + [[nodiscard]] Expected> numberPath(ProtoFieldId field_id) const { + if (field_id >= requested_.size()) { + return Status::error("protobuf field id is outside the traversal plan"); + } + PJ_PARSER_MODULE_TRY { + return requested_[field_id].numbers; + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while copying a protobuf field path"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while copying a protobuf field path"); + } + } + + [[nodiscard]] Expected locate(const ProtoReader& reader, ProtoFieldId field_id) const { + if (field_id >= requested_.size()) { + return Status::error("protobuf field id is outside the traversal plan"); + } + const auto& numbers = requested_[field_id].numbers; + if (numbers.size() > ProtoReader::kMaxRecursionDepth) { + return Status::error("protobuf field path depth exceeds 64"); + } + ProtoReader current = reader; + for (size_t index = 0; index < numbers.size(); ++index) { + auto field = current.last(numbers[index]); + if (!field) { + return field.status(); + } + if (index + 1 == numbers.size()) { + return field; + } + if (field->wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("protobuf field-path intermediate is not a message"); + } + current = ProtoReader(field->bytes); + } + return Status::error("protobuf field path is empty"); + } + + private: + std::vector requested_; + + friend class ProtoFieldLocator; +}; + +class ProtoFieldLocator { + public: + explicit ProtoFieldLocator(ByteView descriptor_set, std::string_view root_type = {}) { + PJ_PARSER_MODULE_TRY { + status_ = parse(descriptor_set, root_type); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + status_ = Status::error("allocation failed while decoding FileDescriptorSet"); + } + PJ_PARSER_MODULE_CATCH_ALL { + status_ = Status::error("unexpected failure while decoding FileDescriptorSet"); + } + } + + [[nodiscard]] const Status& status() const noexcept { + return status_; + } + + [[nodiscard]] Expected locate(std::initializer_list paths) const { + PJ_PARSER_MODULE_TRY { + return locate(std::vector(paths)); + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while compiling protobuf field paths"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while compiling protobuf field paths"); + } + } + + [[nodiscard]] Expected locate(const std::vector& paths) const { + if (!status_.isOk()) { + return status_; + } + PJ_PARSER_MODULE_TRY { + ProtoTraversalPlan plan; + for (const auto& path : paths) { + auto request = compilePath(path); + if (!request) { + return request.status(); + } + for (const auto& existing : plan.requested_) { + if (existing.path == request->path) { + return Status::error("duplicate protobuf field path requested"); + } + } + plan.requested_.push_back(std::move(*request)); + } + if (plan.requested_.empty()) { + return Status::error("at least one protobuf field path is required"); + } + return plan; + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while compiling protobuf field paths"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while compiling protobuf field paths"); + } + } + + private: + [[nodiscard]] Status parse(ByteView descriptor_set, std::string_view root_type) { + if (descriptor_set.data == nullptr || descriptor_set.size == 0) { + return Status::error("FileDescriptorSet is empty"); + } + ProtoReader set_reader(descriptor_set); + auto files = set_reader.matching(1); + if (!files || files->empty()) { + return files ? Status::error("FileDescriptorSet contains no files") : files.status(); + } + for (const auto& file_field : *files) { + if (file_field.wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("FileDescriptorSet file entry is not a message"); + } + Status decoded = parseFile(file_field.bytes); + if (!decoded.isOk()) { + return decoded; + } + } + + if (root_type.empty()) { + size_t root_count = 0; + for (size_t index = 0; index < messages_.size(); ++index) { + if (messages_[index].top_level) { + root_index_ = index; + ++root_count; + } + } + if (root_count != 1) { + return Status::error("protobuf root type is required when FileDescriptorSet has multiple top-level messages"); + } + return Status::ok(); + } + + const std::string normalized = detail::normalizeProtoTypeName(root_type); + auto root = findMessage(normalized); + if (!root) { + return root.status(); + } + root_index_ = *root; + return Status::ok(); + } + + [[nodiscard]] Status parseFile(ByteView bytes) { + ProtoReader file(bytes); + std::string package; + auto packages = file.matching(2); + if (!packages) { + return packages.status(); + } + if (!packages->empty()) { + const auto& field = packages->back(); + if (field.wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("FileDescriptorProto package is not a string"); + } + package = detail::protoString(field.bytes); + } + auto declarations = file.matching(4); + if (!declarations) { + return declarations.status(); + } + for (const auto& declaration : *declarations) { + if (declaration.wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("FileDescriptorProto message_type is not a message"); + } + Status decoded = parseMessage(declaration.bytes, package, true, 1); + if (!decoded.isOk()) { + return decoded; + } + } + return Status::ok(); + } + + [[nodiscard]] Status parseMessage(ByteView bytes, const std::string& parent, bool top_level, size_t depth) { + if (depth > ProtoReader::kMaxRecursionDepth) { + return Status::error("protobuf descriptor nesting depth exceeds 64"); + } + ProtoReader message(bytes); + auto name_field = message.last(1); + if (!name_field || name_field->wire_type != ProtoReader::WireType::kLengthDelimited || name_field->bytes.empty()) { + return Status::error("DescriptorProto is missing its name"); + } + const std::string name = detail::protoString(name_field->bytes); + const std::string full_name = parent.empty() ? name : parent + "." + name; + for (const auto& existing : messages_) { + if (existing.full_name == full_name) { + return Status::error("FileDescriptorSet contains a duplicate message name"); + } + } + const size_t message_index = messages_.size(); + messages_.push_back(detail::ProtoSchemaMessage{full_name, {}, top_level}); + + auto fields = message.matching(2); + if (!fields) { + return fields.status(); + } + for (const auto& field : *fields) { + if (field.wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("DescriptorProto field entry is not a message"); + } + auto decoded = parseField(field.bytes); + if (!decoded) { + return decoded.status(); + } + messages_[message_index].fields.push_back(std::move(*decoded)); + } + auto& decoded_fields = messages_[message_index].fields; + std::sort(decoded_fields.begin(), decoded_fields.end(), [](const auto& lhs, const auto& rhs) { + return lhs.name < rhs.name; + }); + for (size_t index = 1; index < decoded_fields.size(); ++index) { + if (decoded_fields[index - 1].name == decoded_fields[index].name) { + return Status::error("DescriptorProto contains a duplicate field name or number"); + } + } + std::sort(decoded_fields.begin(), decoded_fields.end(), [](const auto& lhs, const auto& rhs) { + return lhs.number < rhs.number; + }); + for (size_t index = 1; index < decoded_fields.size(); ++index) { + if (decoded_fields[index - 1].number == decoded_fields[index].number) { + return Status::error("DescriptorProto contains a duplicate field name or number"); + } + } + + auto nested = message.matching(3); + if (!nested) { + return nested.status(); + } + for (const auto& declaration : *nested) { + if (declaration.wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("DescriptorProto nested_type is not a message"); + } + Status decoded = parseMessage(declaration.bytes, full_name, false, depth + 1); + if (!decoded.isOk()) { + return decoded; + } + } + return Status::ok(); + } + + [[nodiscard]] Expected parseField(ByteView bytes) const { + ProtoReader field(bytes); + auto name = field.last(1); + auto number = field.varint(3); + auto label = field.varint(4); + auto type = field.varint(5); + if (!name || name->wire_type != ProtoReader::WireType::kLengthDelimited || name->bytes.empty() || !number || + !label || !type) { + return Status::error("FieldDescriptorProto is missing a required field"); + } + if (*number == 0 || *number > UINT32_C(0x1FFFFFFF) || *label == 0 || *label > 3 || *type == 0 || *type > 18 || + *type == 10) { + return Status::error("FieldDescriptorProto uses an unsupported field number, label, or type"); + } + detail::ProtoSchemaField result; + result.name = detail::protoString(name->bytes); + result.number = static_cast(*number); + result.type = static_cast(*type); + result.repeated = *label == 3; + if (result.type == 11) { + auto type_name = field.last(6); + if (!type_name || type_name->wire_type != ProtoReader::WireType::kLengthDelimited) { + return Status::error("message FieldDescriptorProto is missing type_name"); + } + result.type_name = detail::normalizeProtoTypeName(detail::protoString(type_name->bytes)); + } + return result; + } + + [[nodiscard]] Expected findMessage(std::string_view full_name) const { + for (size_t index = 0; index < messages_.size(); ++index) { + if (messages_[index].full_name == full_name) { + return index; + } + } + return Status::error("protobuf message type is absent from FileDescriptorSet"); + } + + [[nodiscard]] Expected compilePath(std::string_view path) const { + if (path.empty()) { + return Status::error("protobuf field path is empty"); + } + detail::ProtoRequestedField request; + request.path = std::string(path); + size_t message_index = root_index_; + size_t position = 0; + while (position < path.size()) { + const size_t dot = path.find('.', position); + const std::string_view component = + path.substr(position, dot == std::string_view::npos ? path.size() - position : dot - position); + if (component.empty()) { + return Status::error("protobuf field path contains an empty component"); + } + const detail::ProtoSchemaField* selected = nullptr; + for (const auto& field : messages_[message_index].fields) { + if (field.name == component) { + selected = &field; + break; + } + } + if (selected == nullptr) { + return Status::error("protobuf field path is absent from FileDescriptorSet"); + } + request.numbers.push_back(selected->number); + if (dot == std::string_view::npos) { + return request; + } + if (selected->type != 11 || selected->repeated) { + return Status::error("protobuf field path may descend only through singular message fields"); + } + auto nested = findMessage(selected->type_name); + if (!nested) { + return nested.status(); + } + message_index = *nested; + position = dot + 1; + if (request.numbers.size() >= ProtoReader::kMaxRecursionDepth) { + return Status::error("protobuf field path depth exceeds 64"); + } + } + return Status::error("protobuf field path is malformed"); + } + + std::vector messages_; + size_t root_index_ = 0; + Status status_; +}; + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/proto_reader.hpp b/pj_base/include/pj_base/parser_module/proto_reader.hpp new file mode 100644 index 00000000..361cd7c3 --- /dev/null +++ b/pj_base/include/pj_base/parser_module/proto_reader.hpp @@ -0,0 +1,447 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** @file proto_reader.hpp @brief Bounds-checked protobuf wire reader. */ + +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/core.hpp" + +namespace pj { + +class ProtoReader { + public: + static constexpr size_t kMaxRecursionDepth = 64; + + enum class WireType : uint8_t { + kVarint = 0, + kFixed64 = 1, + kLengthDelimited = 2, + kStartGroup = 3, + kEndGroup = 4, + kFixed32 = 5, + }; + + struct Field { + uint32_t number = 0; + WireType wire_type = WireType::kVarint; + uint64_t integer = 0; + ByteView bytes; + }; + + /// Nothrow-owned match collection. A lookup retains a bounded number of + /// occurrences; larger valid messages report a resource error instead of + /// exhausting guest memory. + class FieldList { + public: + static constexpr size_t kMaximumFields = 64 * 1024; + + FieldList() = default; + ~FieldList() { + delete[] fields_; + } + FieldList(FieldList&& other) noexcept + : fields_(std::exchange(other.fields_, nullptr)), + size_(std::exchange(other.size_, 0)), + capacity_(std::exchange(other.capacity_, 0)) {} + FieldList& operator=(FieldList&& other) noexcept { + if (this != &other) { + delete[] fields_; + fields_ = std::exchange(other.fields_, nullptr); + size_ = std::exchange(other.size_, 0); + capacity_ = std::exchange(other.capacity_, 0); + } + return *this; + } + FieldList(const FieldList&) = delete; + FieldList& operator=(const FieldList&) = delete; + + [[nodiscard]] bool empty() const noexcept { + return size_ == 0; + } + [[nodiscard]] size_t size() const noexcept { + return size_; + } + [[nodiscard]] const Field& back() const noexcept { + return fields_[size_ - 1]; + } + [[nodiscard]] const Field* begin() const noexcept { + return fields_; + } + [[nodiscard]] const Field* end() const noexcept { + return fields_ == nullptr ? nullptr : fields_ + size_; + } + + [[nodiscard]] Status push(Field field) noexcept { + if (size_ == kMaximumFields) { + return Status::error("protobuf field match count exceeds the configured limit"); + } + if (size_ == capacity_) { + const size_t next_capacity = capacity_ == 0 ? size_t{8} : capacity_ * 2; + const size_t bounded_capacity = next_capacity < kMaximumFields ? next_capacity : kMaximumFields; + auto* grown = new (std::nothrow) Field[bounded_capacity]; + if (grown == nullptr) { + return Status::error("allocation failed while collecting protobuf fields"); + } + for (size_t index = 0; index < size_; ++index) { + grown[index] = fields_[index]; + } + delete[] fields_; + fields_ = grown; + capacity_ = bounded_capacity; + } + fields_[size_++] = field; + return Status::ok(); + } + + private: + Field* fields_ = nullptr; + size_t size_ = 0; + size_t capacity_ = 0; + }; + + explicit ProtoReader(ByteView message) : message_(message) {} + + [[nodiscard]] Expected last(uint32_t field_number) const { + auto fields = matching(field_number); + if (!fields) { + return fields.status(); + } + if (fields->empty()) { + return Status::error("protobuf field is absent"); + } + return fields->back(); + } + + [[nodiscard]] Expected matching(uint32_t field_number) const { + if (field_number == 0 || field_number > UINT32_C(0x1FFFFFFF)) { + return Status::error("protobuf field number is invalid"); + } + FieldList result; + size_t position = 0; + while (position < message_.size) { + auto field = nextField(position, depth_); + if (!field) { + return field.status(); + } + if (field->number == field_number) { + Status retained = result.push(*field); + if (!retained.isOk()) { + return retained; + } + } + } + return result; + } + + [[nodiscard]] Expected varint(uint32_t field_number) const { + auto field = last(field_number); + if (!field) { + return field.status(); + } + if (field->wire_type != WireType::kVarint) { + return Status::error("protobuf field is not a varint"); + } + return field->integer; + } + + [[nodiscard]] Expected zigzag(uint32_t field_number) const { + auto value = varint(field_number); + if (!value) { + return value.status(); + } + const uint64_t sign = UINT64_C(0) - (*value & 1U); + const uint64_t bits = (*value >> 1U) ^ sign; + int64_t result = 0; + std::memcpy(&result, &bits, sizeof(result)); + return result; + } + + [[nodiscard]] Expected fixed32(uint32_t field_number) const { + auto field = last(field_number); + if (!field) { + return field.status(); + } + if (field->wire_type != WireType::kFixed32) { + return Status::error("protobuf field is not fixed32"); + } + return static_cast(field->integer); + } + + [[nodiscard]] Expected fixed64(uint32_t field_number) const { + auto field = last(field_number); + if (!field) { + return field.status(); + } + if (field->wire_type != WireType::kFixed64) { + return Status::error("protobuf field is not fixed64"); + } + return field->integer; + } + + [[nodiscard]] Expected bytes(uint32_t field_number) const { + auto field = last(field_number); + if (!field) { + return field.status(); + } + if (field->wire_type != WireType::kLengthDelimited) { + return Status::error("protobuf field is not length-delimited"); + } + return field->bytes; + } + + [[nodiscard]] Expected submessage(uint32_t field_number) const { + if (depth_ == kMaxRecursionDepth) { + return Status::error("protobuf recursion depth exceeds 64"); + } + auto value = bytes(field_number); + if (!value) { + return value.status(); + } + return ProtoReader(*value, depth_ + 1); + } + + /// Collect a repeated integer field. Both unpacked varints and packed + /// length-delimited varint payloads are accepted in encounter order. + [[nodiscard]] Expected> repeatedVarints(uint32_t field_number) const { + auto fields = matching(field_number); + if (!fields) { + return fields.status(); + } + PJ_PARSER_MODULE_TRY { + std::vector values; + for (const auto& field : *fields) { + if (field.wire_type == WireType::kVarint) { + values.push_back(field.integer); + continue; + } + if (field.wire_type != WireType::kLengthDelimited) { + return Status::error("repeated protobuf integer uses an incompatible wire type"); + } + size_t position = 0; + while (position < field.bytes.size) { + auto value = readVarint(field.bytes, position); + if (!value) { + return value.status(); + } + values.push_back(*value); + } + } + return values; + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while collecting repeated protobuf values"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while collecting repeated protobuf values"); + } + } + + [[nodiscard]] Expected> repeatedFixed32(uint32_t field_number) const { + return repeatedFixed(field_number, WireType::kFixed32, 4); + } + + [[nodiscard]] Expected> repeatedFixed64(uint32_t field_number) const { + return repeatedFixed(field_number, WireType::kFixed64, 8); + } + + private: + ProtoReader(ByteView message, size_t depth) : message_(message), depth_(depth) {} + + [[nodiscard]] static Expected readVarint(ByteView bytes, size_t& position) { + if (bytes.data == nullptr && bytes.size != 0) { + return Status::error("protobuf input storage is null"); + } + uint64_t value = 0; + for (size_t byte_index = 0; byte_index < 10; ++byte_index) { + if (position >= bytes.size) { + return Status::error("truncated protobuf varint"); + } + const uint8_t byte = bytes.data[position++]; + if (byte_index == 9 && (byte & UINT8_C(0xFE)) != 0) { + return Status::error("protobuf varint overflows uint64"); + } + value |= static_cast(byte & UINT8_C(0x7F)) << (byte_index * 7); + if ((byte & UINT8_C(0x80)) == 0) { + return value; + } + } + return Status::error("protobuf varint exceeds ten bytes"); + } + + [[nodiscard]] Expected nextField(size_t& position, size_t depth) const { + auto key = readVarint(message_, position); + if (!key) { + return key.status(); + } + const uint64_t raw_field_number = *key >> 3U; + const uint32_t field_number = static_cast(raw_field_number); + const uint8_t wire_value = static_cast(*key & 7U); + if (raw_field_number == 0 || raw_field_number > UINT32_C(0x1FFFFFFF)) { + return Status::error("protobuf field number is invalid"); + } + if (wire_value > static_cast(WireType::kFixed32) || wire_value == 6 || wire_value == 7) { + return Status::error("protobuf wire type is invalid"); + } + const auto wire_type = static_cast(wire_value); + Field field; + field.number = field_number; + field.wire_type = wire_type; + switch (wire_type) { + case WireType::kVarint: { + auto value = readVarint(message_, position); + if (!value) { + return value.status(); + } + field.integer = *value; + return field; + } + case WireType::kFixed64: + if (position > message_.size || 8 > message_.size - position) { + return Status::error("truncated protobuf fixed64 field"); + } + field.integer = readLittleEndian(message_.data + position, 8); + position += 8; + return field; + case WireType::kLengthDelimited: { + auto length = readVarint(message_, position); + if (!length) { + return length.status(); + } + if (*length > std::numeric_limits::max() || position > message_.size || + static_cast(*length) > message_.size - position) { + return Status::error("protobuf length-delimited field exceeds the remaining input"); + } + field.bytes = ByteView(message_.data + position, static_cast(*length)); + position += static_cast(*length); + return field; + } + case WireType::kStartGroup: { + Status skipped = skipGroup(position, field_number, depth + 1); + if (!skipped.isOk()) { + return skipped; + } + return field; + } + case WireType::kEndGroup: + return Status::error("unexpected protobuf end-group tag"); + case WireType::kFixed32: + if (position > message_.size || 4 > message_.size - position) { + return Status::error("truncated protobuf fixed32 field"); + } + field.integer = readLittleEndian(message_.data + position, 4); + position += 4; + return field; + } + return Status::error("protobuf wire type is invalid"); + } + + [[nodiscard]] Status skipGroup(size_t& position, uint32_t group_number, size_t depth) const { + if (depth > kMaxRecursionDepth) { + return Status::error("protobuf recursion depth exceeds 64"); + } + while (position < message_.size) { + auto key = readVarint(message_, position); + if (!key) { + return key.status(); + } + const uint64_t raw_field_number = *key >> 3U; + const uint8_t wire_value = static_cast(*key & 7U); + if (raw_field_number == 0 || raw_field_number > UINT32_C(0x1FFFFFFF) || wire_value > 5 || wire_value == 6 || + wire_value == 7) { + return Status::error("invalid protobuf field inside a group"); + } + const uint32_t field_number = static_cast(raw_field_number); + const auto wire_type = static_cast(wire_value); + if (wire_type == WireType::kEndGroup) { + return field_number == group_number ? Status::ok() : Status::error("protobuf group end tag does not match"); + } + if (wire_type == WireType::kStartGroup) { + Status nested = skipGroup(position, field_number, depth + 1); + if (!nested.isOk()) { + return nested; + } + continue; + } + Status skipped = skipValue(position, wire_type); + if (!skipped.isOk()) { + return skipped; + } + } + return Status::error("truncated protobuf group"); + } + + [[nodiscard]] Status skipValue(size_t& position, WireType wire_type) const { + if (wire_type == WireType::kVarint) { + auto value = readVarint(message_, position); + return value ? Status::ok() : value.status(); + } + if (wire_type == WireType::kFixed64 || wire_type == WireType::kFixed32) { + const size_t width = wire_type == WireType::kFixed64 ? 8 : 4; + if (position > message_.size || width > message_.size - position) { + return Status::error("truncated protobuf fixed field"); + } + position += width; + return Status::ok(); + } + if (wire_type == WireType::kLengthDelimited) { + auto length = readVarint(message_, position); + if (!length || *length > std::numeric_limits::max() || position > message_.size || + static_cast(*length) > message_.size - position) { + return Status::error("truncated protobuf length-delimited field"); + } + position += static_cast(*length); + return Status::ok(); + } + return Status::error("invalid protobuf group field"); + } + + [[nodiscard]] static uint64_t readLittleEndian(const uint8_t* data, size_t width) { + uint64_t value = 0; + for (size_t index = 0; index < width; ++index) { + value |= static_cast(data[index]) << (index * 8); + } + return value; + } + + template + [[nodiscard]] Expected> repeatedFixed( + uint32_t field_number, WireType unpacked_type, size_t width) const { + auto fields = matching(field_number); + if (!fields) { + return fields.status(); + } + PJ_PARSER_MODULE_TRY { + std::vector values; + for (const auto& field : *fields) { + if (field.wire_type == unpacked_type) { + values.push_back(static_cast(field.integer)); + continue; + } + if (field.wire_type != WireType::kLengthDelimited || field.bytes.size % width != 0) { + return Status::error("repeated protobuf fixed field uses an incompatible wire encoding"); + } + for (size_t offset = 0; offset < field.bytes.size; offset += width) { + values.push_back(static_cast(readLittleEndian(field.bytes.data + offset, width))); + } + } + return values; + } + PJ_PARSER_MODULE_CATCH_BAD_ALLOC { + return Status::error("allocation failed while collecting repeated protobuf fixed values"); + } + PJ_PARSER_MODULE_CATCH_ALL { + return Status::error("unexpected failure while collecting repeated protobuf fixed values"); + } + } + + ByteView message_; + size_t depth_ = 0; +}; + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module/time.hpp b/pj_base/include/pj_base/parser_module/time.hpp new file mode 100644 index 00000000..b0671b4a --- /dev/null +++ b/pj_base/include/pj_base/parser_module/time.hpp @@ -0,0 +1,41 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** @file time.hpp @brief Checked ROS and protobuf timestamp normalization. */ + +#include +#include + +#include "pj_base/parser_module/core.hpp" + +namespace pj { +namespace detail { + +inline Expected combineSecondsAndNanos(int64_t seconds, int32_t nanos) { + constexpr int64_t kNanosPerSecond = INT64_C(1000000000); + if (nanos < 0 || nanos >= kNanosPerSecond) { + return Status::error("timestamp nanoseconds are outside [0, 1000000000)"); + } + const int64_t positive_room = (std::numeric_limits::max() - nanos) / kNanosPerSecond; + const int64_t negative_room = std::numeric_limits::min() / kNanosPerSecond; + if (seconds > positive_room || seconds < negative_room) { + return Status::error("timestamp is outside the int64 nanosecond range"); + } + return seconds * kNanosPerSecond + nanos; +} + +} // namespace detail + +[[nodiscard]] inline Expected readRosTime(int32_t seconds, uint32_t nanoseconds) { + if (nanoseconds >= UINT32_C(1000000000)) { + return Status::error("ROS time nanoseconds are outside [0, 1000000000)"); + } + return detail::combineSecondsAndNanos(seconds, static_cast(nanoseconds)); +} + +[[nodiscard]] inline Expected readProtoTimestamp(int64_t seconds, int32_t nanoseconds) { + return detail::combineSecondsAndNanos(seconds, nanoseconds); +} + +} // namespace pj diff --git a/pj_base/include/pj_base/parser_module_abi.h b/pj_base/include/pj_base/parser_module_abi.h new file mode 100644 index 00000000..61a39463 --- /dev/null +++ b/pj_base/include/pj_base/parser_module_abi.h @@ -0,0 +1,180 @@ +/** + * @file parser_module_abi.h + * @brief Frozen parser-module export ABI and host-side byte codecs. + * + * Native and wasm parser modules expose the same operational functions. Every + * address-like value is a uint64_t module-space token: a process address for + * native artifacts or a linear-memory offset for wasm artifacts. The C++ + * helpers below encode and decode the little-endian blocks exchanged through + * those functions without exposing C++ objects across the module boundary. + */ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#ifndef PJ_PARSER_MODULE_ABI_H +#define PJ_PARSER_MODULE_ABI_H + +#include + +#define PJ_PARSER_MODULE_ABI_VERSION UINT32_C(1) + +/* Result codes are append-only. DECLINE is valid only from bind; parse uses + * OK or a negative error. Every negative result records a UTF-8 diagnostic. */ +#define PJ_MODULE_OK INT32_C(0) +#define PJ_MODULE_DECLINE INT32_C(1) +#define PJ_MODULE_ERR_GENERIC (-INT32_C(1)) +#define PJ_MODULE_ERR_BAD_TOKEN (-INT32_C(2)) +#define PJ_MODULE_ERR_MALFORMED_INPUT (-INT32_C(3)) +#define PJ_MODULE_ERR_BAD_CLAIM_INDEX (-INT32_C(4)) +#define PJ_MODULE_ERR_ALLOCATION_FAILURE (-INT32_C(5)) + +#define PJ_MODULE_ABI_EXPORT_NAME "pj_module_abi" +#define PJ_MODULE_CREATE_EXPORT_NAME "pj_module_create" +#define PJ_MODULE_DESTROY_EXPORT_NAME "pj_module_destroy" +#define PJ_MODULE_BIND_EXPORT_NAME "pj_module_bind" +#define PJ_MODULE_PARSE_EXPORT_NAME "pj_module_parse" +#define PJ_MODULE_LAST_ERROR_EXPORT_NAME "pj_module_last_error" +#define PJ_MODULE_ALLOC_EXPORT_NAME "pj_module_alloc" +#define PJ_MODULE_FREE_EXPORT_NAME "pj_module_free" +#define PJ_MODULE_MANIFEST_ADDR_EXPORT_NAME "pj_module_manifest_addr" +#define PJ_MODULE_MANIFEST_LEN_EXPORT_NAME "pj_module_manifest_len" + +#define PJ_PARSER_MODULE_MANIFEST_SECTION_NAME "pj_parser_module_manifest" +#define PJ_MODULE_ERROR_BUFFER_SIZE UINT32_C(512) +/* Instance token zero is never valid. Creation failures are retrieved by + * passing this token to pj_module_last_error. */ +#define PJ_MODULE_CREATION_ERROR_TOKEN UINT64_C(0) + +#define PJ_MODULE_BINDING_INFO_VERSION_V1 UINT16_C(1) +#define PJ_MODULE_OUTPUT_DESCRIPTOR_VERSION_V1 UINT16_C(1) +#define PJ_MODULE_ROUTE_SCALAR UINT8_C(1) +#define PJ_MODULE_ROUTE_OBJECT UINT8_C(2) +#define PJ_MODULE_PARSE_INPUT_FLAG_HAS_TIMESTAMP UINT8_C(1) +#define PJ_MODULE_SCALAR_VALUE_F64 UINT8_C(0) +#define PJ_MODULE_SCALAR_VALUE_I64 UINT8_C(1) +#define PJ_MODULE_SCALAR_VALUE_U64 UINT8_C(2) +#define PJ_MODULE_SCALAR_VALUE_BOOL UINT8_C(3) +#define PJ_MODULE_SCALAR_VALUE_STRING UINT8_C(4) + +#ifdef __cplusplus +extern "C" { +#endif + +/** Operational export signatures, resolved by the matching *_EXPORT_NAME. + * + * Lifecycle is create(claim_index), bind(BindingInfo), any number of parse + * calls, then destroy. Module-owned output descriptors returned by parse stay + * valid until the next call on that instance or destroy. Native addresses are + * process pointers encoded as uint64_t; wasm addresses are linear-memory + * offsets and are never host pointers. The host serializes lifecycle calls. + */ +typedef uint32_t (*PJ_module_abi_fn_t)(void); +typedef uint64_t (*PJ_module_create_fn_t)(uint32_t claim_index); +typedef void (*PJ_module_destroy_fn_t)(uint64_t inst); +typedef int32_t (*PJ_module_bind_fn_t)(uint64_t inst, uint64_t info_addr, uint64_t info_len); +typedef int32_t (*PJ_module_parse_fn_t)( + uint64_t inst, uint64_t in_addr, uint64_t in_len, uint64_t out_addr_ptr, uint64_t out_len_ptr); +typedef uint64_t (*PJ_module_last_error_fn_t)(uint64_t inst, uint64_t buf_addr, uint64_t buf_cap); +typedef uint64_t (*PJ_module_alloc_fn_t)(uint64_t size); +typedef void (*PJ_module_free_fn_t)(uint64_t addr, uint64_t size); + +/** Native-only metadata exports. Wasm modules deliver the manifest only in + * PJ_PARSER_MODULE_MANIFEST_SECTION_NAME and must not export these functions. + */ +typedef uint64_t (*PJ_module_manifest_addr_fn_t)(void); +typedef uint64_t (*PJ_module_manifest_len_fn_t)(void); + +#ifdef __cplusplus +} + +#include +#include +#include +#include + +#include "pj_base/expected.hpp" +#include "pj_base/span.hpp" + +namespace PJ::parser_module { + +inline constexpr uint16_t kBindingInfoVersionV1 = PJ_MODULE_BINDING_INFO_VERSION_V1; +inline constexpr uint16_t kOutputDescriptorVersionV1 = PJ_MODULE_OUTPUT_DESCRIPTOR_VERSION_V1; + +enum class Route : uint8_t { + kScalar = PJ_MODULE_ROUTE_SCALAR, + kObject = PJ_MODULE_ROUTE_OBJECT, +}; + +/** BindingInfo v1 fields. All views are borrowed from the caller on write and + * from the encoded block on read. + */ +struct BindingInfoV1 { + Route route = Route::kScalar; + uint32_t claim_index = 0; + uint16_t expected_object_type = 0; + Span encoding; + Span type_name; + Span schema; + Span claim_id; + Span config_json; + Span schema_digest; +}; + +/** Parse-input framing. `payload` is borrowed from the caller or input block. */ +struct ParseInputV1 { + bool has_timestamp = false; + int64_t timestamp_ns = 0; + Span payload; +}; + +struct ObjectSpliceV1 { + uint32_t field_number = 0; + uint64_t input_offset = 0; + uint64_t input_length = 0; +}; + +/** Object output descriptor. `wire` is full canonical wire without a splice, + * or partial canonical wire with the optional eligible bulk field elided. + */ +struct ObjectOutputV1 { + uint16_t object_type = 0; + std::optional splice; + Span wire; +}; + +using ScalarValueV1 = std::variant; + +struct ScalarFieldV1 { + /** Name offsets in encoded descriptors are relative to byte zero of the + * complete output block. Writers place names after all field values. + */ + std::string_view name; + ScalarValueV1 value; +}; + +struct ScalarOutputV1 { + bool has_timestamp = false; + int64_t timestamp_ns = 0; + std::vector fields; +}; + +using OutputDescriptorV1 = std::variant; + +/** Encode/decode the frozen little-endian module blocks. Readers return + * borrowed views into `bytes` and reject malformed, out-of-range, truncated, + * or unsupported-version data. + */ +[[nodiscard]] Expected> writeBindingInfoV1(const BindingInfoV1& info); +[[nodiscard]] Expected readBindingInfoV1(Span bytes); + +[[nodiscard]] Expected> writeParseInputV1(const ParseInputV1& input); +[[nodiscard]] Expected readParseInputV1(Span bytes); + +[[nodiscard]] Expected> writeOutputDescriptorV1(const OutputDescriptorV1& output); +[[nodiscard]] Expected readOutputDescriptorV1(Span bytes); + +} // namespace PJ::parser_module + +#endif + +#endif // PJ_PARSER_MODULE_ABI_H diff --git a/pj_base/include/pj_base/parser_module_manifest.hpp b/pj_base/include/pj_base/parser_module_manifest.hpp new file mode 100644 index 00000000..e1b3427b --- /dev/null +++ b/pj_base/include/pj_base/parser_module_manifest.hpp @@ -0,0 +1,37 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file parser_module_manifest.hpp + * @brief WebAssembly parser-module manifest custom-section codec. + */ + +#include +#include + +#include "pj_base/expected.hpp" +#include "pj_base/parser_module_abi.h" +#include "pj_base/span.hpp" + +namespace PJ::parser_module { + +/** Append the parser-module manifest custom section to a wasm binary. + * + * The input must have a valid wasm preamble and fully bounded sections. It + * must not already contain PJ_PARSER_MODULE_MANIFEST_SECTION_NAME. The result + * contains exactly one such section, appended after every input section, and + * its payload after the custom-section name is byte-identical to `manifest`. + */ +[[nodiscard]] Expected> appendManifestSection( + Span wasm, Span manifest); + +/** Return the borrowed manifest bytes from a wasm custom section. + * + * The complete module is bounds-checked. A malformed preamble or section, no + * parser-module manifest section, or more than one such section is an error. + * The returned view remains valid only while `wasm` remains valid. + */ +[[nodiscard]] Expected> readManifestSection(Span wasm); + +} // namespace PJ::parser_module diff --git a/pj_base/include/pj_base/parser_route_claims_protocol.h b/pj_base/include/pj_base/parser_route_claims_protocol.h new file mode 100644 index 00000000..adb30397 --- /dev/null +++ b/pj_base/include/pj_base/parser_route_claims_protocol.h @@ -0,0 +1,75 @@ +/** + * @file parser_route_claims_protocol.h + * @brief Additive C ABI extension for exact MessageParser route claims. + * + * A MessageParser built with a route-aware SDK exposes this table from + * get_plugin_extension("pj.parser_route_claims.v1"). Classification reports + * exact handler-table coverage only. The host owns wildcard claims and never + * asks this extension to report them. + */ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#ifndef PJ_PARSER_ROUTE_CLAIMS_PROTOCOL_H +#define PJ_PARSER_ROUTE_CLAIMS_PROTOCOL_H + +#include +#include +#include + +#include "pj_base/builtin_object_abi.h" +#include "pj_base/plugin_data_api.h" + +#ifdef __cplusplus +extern "C" { +#endif + +#define PJ_PARSER_ROUTE_CLAIMS_EXTENSION_V1 "pj.parser_route_claims.v1" + +#define PJ_PARSER_ROUTE_FLAG_SCALAR_V1 UINT16_C(1) +#define PJ_PARSER_ROUTE_FLAG_OBJECT_V1 UINT16_C(2) +#define PJ_PARSER_ROUTE_MATCH_EXACT_V1 UINT16_C(0) +#define PJ_PARSER_ROUTE_STATUS_CLAIMED_V1 UINT16_C(0) +#define PJ_PARSER_ROUTE_STATUS_DECLINED_V1 UINT16_C(1) + +/** Exact route classification for one schema type. + * + * `route_flags` uses bit 0 for the scalar route and bit 1 for the object + * route. `match` must be PJ_PARSER_ROUTE_MATCH_EXACT_V1; other values are + * invalid because this extension never reports wildcard claims. `status` is + * claimed or declined. Failures are returned by classify_routes itself and + * are never encoded as a status. `object_type` is NONE unless the object + * route is claimed. + */ +typedef struct PJ_route_classification_v1_t { + uint16_t route_flags; + uint16_t match; + uint16_t status; + uint16_t object_type; +} PJ_route_classification_v1_t; + +/** Route-aware parser classification extension v1. + * + * classify_routes is [thread-safe], pure, and synchronous. It is called + * after bind_schema on the same instance. A successful decline means there + * is no exact handler-table claim for `type_name`; it says nothing about the + * host-owned wildcard scalar claim. On classification failure the provider + * returns false, populates `out_error`, and leaves `out` unspecified. + */ +typedef struct PJ_parser_route_claims_v1_t { + /** sizeof(PJ_parser_route_claims_v1_t) for this append-only table revision. */ + uint32_t struct_size; + bool (*classify_routes)( + void* ctx, PJ_string_view_t type_name, PJ_bytes_view_t schema, PJ_route_classification_v1_t* out, + PJ_error_t* out_error) PJ_NOEXCEPT; +} PJ_parser_route_claims_v1_t; + +#define PJ_PARSER_ROUTE_CLAIMS_V1_MIN_SIZE \ + (offsetof(PJ_parser_route_claims_v1_t, classify_routes) + \ + sizeof(bool (*)(void*, PJ_string_view_t, PJ_bytes_view_t, PJ_route_classification_v1_t*, PJ_error_t*) PJ_NOEXCEPT)) + +#ifdef __cplusplus +} +#endif + +#endif // PJ_PARSER_ROUTE_CLAIMS_PROTOCOL_H diff --git a/pj_base/src/parser_module_abi.cpp b/pj_base/src/parser_module_abi.cpp new file mode 100644 index 00000000..73734c2a --- /dev/null +++ b/pj_base/src/parser_module_abi.cpp @@ -0,0 +1,591 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_base/parser_module_abi.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace PJ::parser_module { +namespace { + +constexpr size_t kBindingHeaderSize = 16; +constexpr size_t kBindingFieldCountV1 = 6; +constexpr size_t kBindingFieldDescriptorSize = 8; +constexpr size_t kParseInputHeaderSize = 24; +constexpr size_t kObjectOutputFixedSize = 36; +constexpr size_t kScalarOutputFixedSize = 24; +constexpr size_t kScalarFieldPrefixSize = 9; + +enum class ScalarValueKind : uint8_t { + kFloat64 = PJ_MODULE_SCALAR_VALUE_F64, + kInt64 = PJ_MODULE_SCALAR_VALUE_I64, + kUint64 = PJ_MODULE_SCALAR_VALUE_U64, + kBool = PJ_MODULE_SCALAR_VALUE_BOOL, + kString = PJ_MODULE_SCALAR_VALUE_STRING, +}; + +template +Expected malformed(std::string message) { + return unexpected(std::string("malformed parser module block: ") + std::move(message)); +} + +[[nodiscard]] bool checkedAdd(size_t lhs, size_t rhs, size_t* out) { + if (rhs > std::numeric_limits::max() - lhs) { + return false; + } + *out = lhs + rhs; + return true; +} + +class ByteWriter { + public: + explicit ByteWriter(size_t reserve) { + bytes_.reserve(reserve); + } + + void u8(uint8_t value) { + bytes_.push_back(value); + } + + void u16(uint16_t value) { + littleEndian(value); + } + + void u32(uint32_t value) { + littleEndian(value); + } + + void u64(uint64_t value) { + littleEndian(value); + } + + void i64(int64_t value) { + u64(std::bit_cast(value)); + } + + void f64(double value) { + u64(std::bit_cast(value)); + } + + void zeros(size_t count) { + bytes_.insert(bytes_.end(), count, uint8_t{0}); + } + + void bytes(Span value) { + if (!value.empty()) { + bytes_.insert(bytes_.end(), value.begin(), value.end()); + } + } + + void string(std::string_view value) { + if (!value.empty()) { + const auto* begin = reinterpret_cast(value.data()); + bytes_.insert(bytes_.end(), begin, begin + value.size()); + } + } + + [[nodiscard]] std::vector finish() && { + return std::move(bytes_); + } + + private: + /// Every v1 block encodes unsigned integers little-endian, least significant + /// byte first. + template + void littleEndian(Unsigned value) { + for (unsigned shift = 0; shift < sizeof(Unsigned) * 8U; shift += 8U) { + bytes_.push_back(static_cast((value >> shift) & UINT64_C(0xFF))); + } + } + + std::vector bytes_; +}; + +class ByteReader { + public: + explicit ByteReader(Span bytes) : bytes_(bytes) {} + + [[nodiscard]] bool u8(uint8_t* out) { + if (remaining() < 1) { + return false; + } + *out = bytes_[position_++]; + return true; + } + + [[nodiscard]] bool u16(uint16_t* out) { + return littleEndian(out); + } + + [[nodiscard]] bool u32(uint32_t* out) { + return littleEndian(out); + } + + [[nodiscard]] bool u64(uint64_t* out) { + return littleEndian(out); + } + + [[nodiscard]] bool i64(int64_t* out) { + uint64_t value = 0; + if (!u64(&value)) { + return false; + } + *out = std::bit_cast(value); + return true; + } + + [[nodiscard]] bool f64(double* out) { + uint64_t value = 0; + if (!u64(&value)) { + return false; + } + *out = std::bit_cast(value); + return true; + } + + [[nodiscard]] bool skip(size_t count) { + if (count > remaining()) { + return false; + } + position_ += count; + return true; + } + + [[nodiscard]] bool take(size_t count, Span* out) { + if (count > remaining()) { + return false; + } + *out = bytes_.subspan(position_, count); + position_ += count; + return true; + } + + [[nodiscard]] size_t remaining() const { + return bytes_.size() - position_; + } + + private: + /// Mirror of ByteWriter::littleEndian: least significant byte first. + template + [[nodiscard]] bool littleEndian(Unsigned* out) { + if (remaining() < sizeof(Unsigned)) { + return false; + } + uint64_t value = 0; + for (unsigned index = 0; index < sizeof(Unsigned); ++index) { + value |= static_cast(bytes_[position_ + index]) << (index * 8U); + } + position_ += sizeof(Unsigned); + *out = static_cast(value); + return true; + } + + Span bytes_; + size_t position_ = 0; +}; + +[[nodiscard]] bool validRoute(uint16_t route) { + return route == static_cast(Route::kScalar) || route == static_cast(Route::kObject); +} + +[[nodiscard]] std::string_view asStringView(Span bytes) { + if (bytes.empty()) { + return {}; + } + return {reinterpret_cast(bytes.data()), bytes.size()}; +} + +[[nodiscard]] Expected scalarValueSize(const ScalarValueV1& value) { + if (std::holds_alternative(value) || std::holds_alternative(value) || + std::holds_alternative(value)) { + return size_t{8}; + } + if (std::holds_alternative(value)) { + return size_t{1}; + } + const auto string_value = std::get(value); + if (string_value.size() > std::numeric_limits::max()) { + return unexpected(std::string("scalar string exceeds the v1 uint32 length limit")); + } + size_t size = 0; + if (!checkedAdd(size_t{4}, string_value.size(), &size)) { + return unexpected(std::string("scalar string size overflows the host size type")); + } + return size; +} + +} // namespace + +Expected> writeBindingInfoV1(const BindingInfoV1& info) { + const uint16_t route = static_cast(info.route); + if (!validRoute(route)) { + return unexpected(std::string("BindingInfo v1 route must be scalar or object")); + } + + const std::array, kBindingFieldCountV1> fields{ + info.encoding, info.type_name, info.schema, info.claim_id, info.config_json, info.schema_digest}; + std::array offsets{}; + size_t total_size = kBindingHeaderSize + kBindingFieldCountV1 * kBindingFieldDescriptorSize; + for (size_t index = 0; index < fields.size(); ++index) { + if (fields[index].size() > std::numeric_limits::max() || + total_size > std::numeric_limits::max()) { + return unexpected(std::string("BindingInfo v1 field exceeds the uint32 offset/length limit")); + } + offsets[index] = static_cast(total_size); + if (!checkedAdd(total_size, fields[index].size(), &total_size)) { + return unexpected(std::string("BindingInfo v1 size overflows the host size type")); + } + } + if (total_size > std::numeric_limits::max()) { + return unexpected(std::string("BindingInfo v1 block exceeds the uint32 offset range")); + } + + ByteWriter writer(total_size); + writer.u16(kBindingInfoVersionV1); + writer.u16(route); + writer.u32(info.claim_index); + writer.u16(info.expected_object_type); + writer.u16(0); + writer.u32(static_cast(kBindingFieldCountV1)); + for (size_t index = 0; index < fields.size(); ++index) { + writer.u32(offsets[index]); + writer.u32(static_cast(fields[index].size())); + } + for (const auto& field : fields) { + writer.bytes(field); + } + return std::move(writer).finish(); +} + +Expected readBindingInfoV1(Span bytes) { + ByteReader reader(bytes); + uint16_t version = 0; + uint16_t route = 0; + uint32_t claim_index = 0; + uint16_t expected_object_type = 0; + uint32_t field_count = 0; + if (!reader.u16(&version) || !reader.u16(&route) || !reader.u32(&claim_index) || !reader.u16(&expected_object_type) || + !reader.skip(2) /* reserved */ || !reader.u32(&field_count)) { + return malformed("truncated BindingInfo v1 header"); + } + if (version != kBindingInfoVersionV1) { + return malformed("unsupported BindingInfo version"); + } + if (!validRoute(route)) { + return malformed("invalid BindingInfo route"); + } + if (field_count < kBindingFieldCountV1) { + return malformed("BindingInfo v1 has fewer than six fields"); + } + if (field_count > reader.remaining() / kBindingFieldDescriptorSize) { + return malformed("truncated BindingInfo field table"); + } + + std::array, kBindingFieldCountV1> fields{}; + for (uint32_t index = 0; index < field_count; ++index) { + uint32_t offset = 0; + uint32_t length = 0; + if (!reader.u32(&offset) || !reader.u32(&length)) { + return malformed("truncated BindingInfo field descriptor"); + } + const size_t field_offset = offset; + const size_t field_length = length; + if (field_offset > bytes.size() || field_length > bytes.size() - field_offset) { + return malformed("BindingInfo field range is outside the block"); + } + if (index < kBindingFieldCountV1) { + fields[index] = bytes.subspan(field_offset, field_length); + } + } + + return BindingInfoV1{ + .route = static_cast(route), + .claim_index = claim_index, + .expected_object_type = expected_object_type, + .encoding = fields[0], + .type_name = fields[1], + .schema = fields[2], + .claim_id = fields[3], + .config_json = fields[4], + .schema_digest = fields[5], + }; +} + +Expected> writeParseInputV1(const ParseInputV1& input) { + size_t total_size = 0; + if (!checkedAdd(kParseInputHeaderSize, input.payload.size(), &total_size)) { + return unexpected(std::string("parse-input v1 size overflows the host size type")); + } + ByteWriter writer(total_size); + writer.u8(input.has_timestamp ? uint8_t{1} : uint8_t{0}); + writer.zeros(7); + writer.i64(input.timestamp_ns); + writer.u64(static_cast(input.payload.size())); + writer.bytes(input.payload); + return std::move(writer).finish(); +} + +Expected readParseInputV1(Span bytes) { + ByteReader reader(bytes); + uint8_t flags = 0; + int64_t timestamp_ns = 0; + uint64_t payload_length = 0; + if (!reader.u8(&flags) || !reader.skip(7) || !reader.i64(×tamp_ns) || !reader.u64(&payload_length)) { + return malformed("truncated parse-input v1 header"); + } + if ((flags & uint8_t{0xFE}) != 0) { + return malformed("parse-input v1 has unknown flag bits"); + } + if (payload_length != reader.remaining()) { + return malformed("parse-input payload length does not match the block"); + } + Span payload; + if (!reader.take(static_cast(payload_length), &payload)) { + return malformed("truncated parse-input payload"); + } + return ParseInputV1{.has_timestamp = (flags & uint8_t{1}) != 0, .timestamp_ns = timestamp_ns, .payload = payload}; +} + +Expected> writeOutputDescriptorV1(const OutputDescriptorV1& output) { + if (const auto* object = std::get_if(&output)) { + size_t total_size = 0; + if (!checkedAdd(kObjectOutputFixedSize, object->wire.size(), &total_size)) { + return unexpected(std::string("object output v1 size overflows the host size type")); + } + ByteWriter writer(total_size); + writer.u16(kOutputDescriptorVersionV1); + writer.u8(static_cast(Route::kObject)); + writer.u8(0); + writer.u16(object->object_type); + writer.u16(object->splice.has_value() ? uint16_t{1} : uint16_t{0}); + writer.u32(object->splice.has_value() ? object->splice->field_number : 0); + writer.u64(object->splice.has_value() ? object->splice->input_offset : 0); + writer.u64(object->splice.has_value() ? object->splice->input_length : 0); + writer.u64(static_cast(object->wire.size())); + writer.bytes(object->wire); + return std::move(writer).finish(); + } + + const auto& scalar = std::get(output); + if (scalar.fields.size() > std::numeric_limits::max()) { + return unexpected(std::string("scalar output v1 has too many fields")); + } + + size_t values_end = kScalarOutputFixedSize; + for (const auto& field : scalar.fields) { + const auto value_size = scalarValueSize(field.value); + if (!value_size) { + return unexpected(value_size.error()); + } + if (field.name.size() > std::numeric_limits::max() || + !checkedAdd(values_end, kScalarFieldPrefixSize, &values_end) || + !checkedAdd(values_end, *value_size, &values_end)) { + return unexpected(std::string("scalar output v1 field size exceeds its encoded range")); + } + } + + std::vector name_offsets; + name_offsets.reserve(scalar.fields.size()); + size_t total_size = values_end; + for (const auto& field : scalar.fields) { + if (total_size > std::numeric_limits::max()) { + return unexpected(std::string("scalar output v1 name offset exceeds uint32")); + } + name_offsets.push_back(static_cast(total_size)); + if (!checkedAdd(total_size, field.name.size(), &total_size) || total_size > std::numeric_limits::max()) { + return unexpected(std::string("scalar output v1 block exceeds the uint32 name-offset range")); + } + } + + ByteWriter writer(total_size); + writer.u16(kOutputDescriptorVersionV1); + writer.u8(static_cast(Route::kScalar)); + writer.u8(0); + writer.u8(scalar.has_timestamp ? uint8_t{1} : uint8_t{0}); + writer.zeros(7); + writer.i64(scalar.timestamp_ns); + writer.u32(static_cast(scalar.fields.size())); + for (size_t index = 0; index < scalar.fields.size(); ++index) { + const auto& field = scalar.fields[index]; + writer.u32(name_offsets[index]); + writer.u32(static_cast(field.name.size())); + if (const auto* float_value = std::get_if(&field.value)) { + writer.u8(static_cast(ScalarValueKind::kFloat64)); + writer.f64(*float_value); + } else if (const auto* signed_value = std::get_if(&field.value)) { + writer.u8(static_cast(ScalarValueKind::kInt64)); + writer.i64(*signed_value); + } else if (const auto* unsigned_value = std::get_if(&field.value)) { + writer.u8(static_cast(ScalarValueKind::kUint64)); + writer.u64(*unsigned_value); + } else if (const auto* bool_value = std::get_if(&field.value)) { + writer.u8(static_cast(ScalarValueKind::kBool)); + writer.u8(*bool_value ? uint8_t{1} : uint8_t{0}); + } else { + const auto string_value = std::get(field.value); + writer.u8(static_cast(ScalarValueKind::kString)); + writer.u32(static_cast(string_value.size())); + writer.string(string_value); + } + } + for (const auto& field : scalar.fields) { + writer.string(field.name); + } + return std::move(writer).finish(); +} + +Expected readOutputDescriptorV1(Span bytes) { + ByteReader reader(bytes); + uint16_t version = 0; + uint8_t route = 0; + if (!reader.u16(&version) || !reader.u8(&route) || !reader.skip(1) /* reserved */) { + return malformed("truncated output descriptor header"); + } + if (version != kOutputDescriptorVersionV1) { + return malformed("unsupported output descriptor version"); + } + + if (route == static_cast(Route::kObject)) { + uint16_t object_type = 0; + uint16_t splice_count = 0; + uint32_t field_number = 0; + uint64_t input_offset = 0; + uint64_t input_length = 0; + uint64_t wire_length = 0; + if (!reader.u16(&object_type) || !reader.u16(&splice_count) || !reader.u32(&field_number) || + !reader.u64(&input_offset) || !reader.u64(&input_length) || !reader.u64(&wire_length)) { + return malformed("truncated object output descriptor"); + } + if (splice_count > 1) { + return malformed("object output splice_count is not 0 or 1"); + } + if (wire_length != reader.remaining()) { + return malformed("object output wire length does not match the block"); + } + Span wire; + if (!reader.take(static_cast(wire_length), &wire)) { + return malformed("truncated object output wire bytes"); + } + std::optional splice; + if (splice_count == 1) { + splice = ObjectSpliceV1{ + .field_number = field_number, + .input_offset = input_offset, + .input_length = input_length, + }; + } + return OutputDescriptorV1(ObjectOutputV1{.object_type = object_type, .splice = splice, .wire = wire}); + } + + if (route != static_cast(Route::kScalar)) { + return malformed("invalid output descriptor route"); + } + + uint8_t has_timestamp = 0; + int64_t timestamp_ns = 0; + uint32_t field_count = 0; + if (!reader.u8(&has_timestamp) || !reader.skip(7) || !reader.i64(×tamp_ns) || !reader.u32(&field_count)) { + return malformed("truncated scalar output descriptor"); + } + if (has_timestamp > 1) { + return malformed("scalar output has_timestamp is not 0 or 1"); + } + if (field_count > reader.remaining() / (kScalarFieldPrefixSize + 1)) { + return malformed("scalar output field count exceeds the block"); + } + + struct DecodedField { + uint32_t name_offset; + uint32_t name_length; + ScalarValueV1 value; + }; + std::vector decoded_fields; + decoded_fields.reserve(field_count); + for (uint32_t index = 0; index < field_count; ++index) { + uint32_t name_offset = 0; + uint32_t name_length = 0; + uint8_t value_kind = 0; + if (!reader.u32(&name_offset) || !reader.u32(&name_length) || !reader.u8(&value_kind)) { + return malformed("truncated scalar field descriptor"); + } + + ScalarValueV1 value; + switch (static_cast(value_kind)) { + case ScalarValueKind::kFloat64: { + double decoded = 0; + if (!reader.f64(&decoded)) { + return malformed("truncated scalar float64 value"); + } + value = decoded; + break; + } + case ScalarValueKind::kInt64: { + int64_t decoded = 0; + if (!reader.i64(&decoded)) { + return malformed("truncated scalar int64 value"); + } + value = decoded; + break; + } + case ScalarValueKind::kUint64: { + uint64_t decoded = 0; + if (!reader.u64(&decoded)) { + return malformed("truncated scalar uint64 value"); + } + value = decoded; + break; + } + case ScalarValueKind::kBool: { + uint8_t decoded = 0; + if (!reader.u8(&decoded) || decoded > 1) { + return malformed("invalid scalar bool value"); + } + value = decoded != 0; + break; + } + case ScalarValueKind::kString: { + uint32_t length = 0; + Span decoded; + if (!reader.u32(&length) || !reader.take(length, &decoded)) { + return malformed("truncated scalar string value"); + } + value = asStringView(decoded); + break; + } + default: + return malformed("unknown scalar value kind"); + } + decoded_fields.push_back(DecodedField{.name_offset = name_offset, .name_length = name_length, .value = value}); + } + + ScalarOutputV1 scalar{ + .has_timestamp = has_timestamp != 0, + .timestamp_ns = timestamp_ns, + .fields = {}, + }; + scalar.fields.reserve(decoded_fields.size()); + for (auto& field : decoded_fields) { + const size_t name_offset = field.name_offset; + const size_t name_length = field.name_length; + if (name_offset > bytes.size() || name_length > bytes.size() - name_offset) { + return malformed("scalar field name range is outside the block"); + } + scalar.fields.push_back( + ScalarFieldV1{ + .name = asStringView(bytes.subspan(name_offset, name_length)), + .value = std::move(field.value), + }); + } + return OutputDescriptorV1(std::move(scalar)); +} + +} // namespace PJ::parser_module diff --git a/pj_base/src/parser_module_manifest.cpp b/pj_base/src/parser_module_manifest.cpp new file mode 100644 index 00000000..3d30601d --- /dev/null +++ b/pj_base/src/parser_module_manifest.cpp @@ -0,0 +1,168 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_base/parser_module_manifest.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace PJ::parser_module { +namespace { + +constexpr std::array kWasmPreamble{0x00, 0x61, 0x73, 0x6D, 0x01, 0x00, 0x00, 0x00}; +constexpr std::string_view kManifestSectionName = PJ_PARSER_MODULE_MANIFEST_SECTION_NAME; + +struct ManifestScan { + size_t count = 0; + Span bytes; +}; + +[[nodiscard]] Expected readVarUint32(Span bytes, size_t* position) { + uint32_t value = 0; + for (size_t index = 0; index < 5; ++index) { + if (*position >= bytes.size()) { + return unexpected(std::string("truncated wasm varuint32")); + } + const uint8_t byte = bytes[(*position)++]; + if (index == 4 && (byte & UINT8_C(0xF0)) != 0) { + return unexpected(std::string("wasm varuint32 overflows uint32")); + } + value |= static_cast(byte & UINT8_C(0x7F)) << (index * 7U); + if ((byte & UINT8_C(0x80)) == 0) { + return value; + } + } + return unexpected(std::string("wasm varuint32 exceeds five bytes")); +} + +void appendVarUint32(std::vector* output, uint32_t value) { + do { + uint8_t byte = static_cast(value & UINT32_C(0x7F)); + value >>= 7U; + if (value != 0) { + byte |= UINT8_C(0x80); + } + output->push_back(byte); + } while (value != 0); +} + +[[nodiscard]] size_t varUint32Size(uint32_t value) { + size_t size = 1; + while (value >= UINT32_C(0x80)) { + value >>= 7U; + ++size; + } + return size; +} + +[[nodiscard]] Expected scanManifestSections(Span wasm) { + if ((wasm.data() == nullptr && !wasm.empty()) || wasm.size() < kWasmPreamble.size() || + !std::equal(kWasmPreamble.begin(), kWasmPreamble.end(), wasm.begin())) { + return unexpected(std::string("invalid or truncated wasm preamble")); + } + + ManifestScan result; + size_t position = kWasmPreamble.size(); + while (position < wasm.size()) { + const uint8_t section_id = wasm[position++]; + auto section_size = readVarUint32(wasm, &position); + if (!section_size) { + return unexpected(section_size.error()); + } + if (static_cast(*section_size) > wasm.size() - position) { + return unexpected(std::string("wasm section exceeds the remaining module bytes")); + } + const size_t section_end = position + *section_size; + if (section_id == 0) { + size_t custom_position = position; + auto name_size = readVarUint32(wasm.first(section_end), &custom_position); + if (!name_size) { + return unexpected(std::string("malformed wasm custom-section name: ") + name_size.error()); + } + if (static_cast(*name_size) > section_end - custom_position) { + return unexpected(std::string("wasm custom-section name exceeds its section")); + } + const auto name = wasm.subspan(custom_position, *name_size); + if (name.size() == kManifestSectionName.size() && + std::equal(name.begin(), name.end(), reinterpret_cast(kManifestSectionName.data()))) { + ++result.count; + const size_t manifest_begin = custom_position + *name_size; + result.bytes = wasm.subspan(manifest_begin, section_end - manifest_begin); + } + } + position = section_end; + } + return result; +} + +} // namespace + +Expected> appendManifestSection(Span wasm, Span manifest) { + auto scanned = scanManifestSections(wasm); + if (!scanned) { + return unexpected(scanned.error()); + } + if (scanned->count != 0) { + return unexpected(std::string("wasm already contains a parser-module manifest section")); + } + if (manifest.data() == nullptr && !manifest.empty()) { + return unexpected(std::string("manifest storage is null")); + } + if (manifest.size() > std::numeric_limits::max()) { + return unexpected(std::string("manifest exceeds the wasm custom-section size range")); + } + + const auto name_size = static_cast(kManifestSectionName.size()); + const uint64_t payload_size_64 = + varUint32Size(name_size) + static_cast(name_size) + static_cast(manifest.size()); + if (payload_size_64 > std::numeric_limits::max() || payload_size_64 > std::numeric_limits::max()) { + return unexpected(std::string("embedded wasm manifest size overflows the output range")); + } + const size_t payload_size = static_cast(payload_size_64); + const size_t section_header_size = 1 + varUint32Size(static_cast(payload_size)); + if (wasm.size() > std::numeric_limits::max() - section_header_size || + payload_size > std::numeric_limits::max() - wasm.size() - section_header_size) { + return unexpected(std::string("embedded wasm manifest size overflows the output range")); + } + + try { + std::vector output; + output.reserve(wasm.size() + section_header_size + payload_size); + output.insert(output.end(), wasm.begin(), wasm.end()); + output.push_back(0); + appendVarUint32(&output, static_cast(payload_size)); + appendVarUint32(&output, name_size); + output.insert(output.end(), kManifestSectionName.begin(), kManifestSectionName.end()); + if (!manifest.empty()) { + output.insert(output.end(), manifest.begin(), manifest.end()); + } + return output; + } catch (const std::bad_alloc&) { + return unexpected(std::string("allocation failed while embedding the wasm manifest")); + } catch (...) { + return unexpected(std::string("unexpected failure while embedding the wasm manifest")); + } +} + +Expected> readManifestSection(Span wasm) { + auto scanned = scanManifestSections(wasm); + if (!scanned) { + return unexpected(scanned.error()); + } + if (scanned->count == 0) { + return unexpected(std::string("wasm has no parser-module manifest section")); + } + if (scanned->count != 1) { + return unexpected(std::string("wasm has multiple parser-module manifest sections")); + } + return scanned->bytes; +} + +} // namespace PJ::parser_module diff --git a/pj_base/tests/abi_layout_sentinels_test.cpp b/pj_base/tests/abi_layout_sentinels_test.cpp index 9bcc9314..845f61f7 100644 --- a/pj_base/tests/abi_layout_sentinels_test.cpp +++ b/pj_base/tests/abi_layout_sentinels_test.cpp @@ -28,11 +28,14 @@ #include #include +#include #include "pj_base/data_source_protocol.h" #include "pj_base/descriptor_import_protocol.h" #include "pj_base/message_parser_protocol.h" #include "pj_base/parser_functional_protocol.h" +#include "pj_base/parser_module_abi.h" +#include "pj_base/parser_route_claims_protocol.h" #include "pj_base/plugin_data_api.h" #include "pj_base/toolbox_protocol.h" @@ -129,6 +132,12 @@ static_assert(PJ_BUILTIN_OBJECT_TYPE_PLOT_MARKERS == 19, "PlotMarkers type id pi static_assert(sizeof(PJ_schema_classification_t) == 4, "PJ_schema_classification_t layout pinned"); static_assert(offsetof(PJ_schema_classification_t, object_type) == 0, "object_type at offset 0"); static_assert(offsetof(PJ_schema_classification_t, reserved) == 2, "reserved at offset 2"); +static_assert(sizeof(PJ_builtin_object_splice_field_v1_t) == 8, "builtin splice table entry size pinned"); +static_assert( + offsetof(PJ_builtin_object_splice_field_v1_t, object_type) == 0, "builtin splice object_type offset pinned"); +static_assert(offsetof(PJ_builtin_object_splice_field_v1_t, reserved) == 2, "builtin splice reserved offset pinned"); +static_assert( + offsetof(PJ_builtin_object_splice_field_v1_t, field_number) == 4, "builtin splice field_number offset pinned"); // Parser functional extension v1. These caller-owned sink tables are the // only values crossing the new parser result boundary; freeze their first @@ -146,8 +155,67 @@ static_assert(offsetof(PJ_parser_functional_v1_t, struct_size) == 0, "functional static_assert(offsetof(PJ_parser_functional_v1_t, parse_scalars) == 8, "functional scalar slot pinned"); static_assert(offsetof(PJ_parser_functional_v1_t, parse_object) == 16, "functional object slot pinned"); static_assert(sizeof(PJ_parser_functional_v1_t) == 24, "functional extension size pinned"); + +static_assert(offsetof(PJ_parser_object_sink_v2_t, struct_size) == 0, "v2 object sink prefix pinned"); +static_assert(offsetof(PJ_parser_object_sink_v2_t, ctx) == 8, "v2 object sink context pinned"); +static_assert(offsetof(PJ_parser_object_sink_v2_t, accept_object) == 16, "v2 full object callback pinned"); +static_assert(offsetof(PJ_parser_object_sink_v2_t, accept_object_spliced) == 24, "v2 splice callback pinned"); +static_assert(sizeof(PJ_parser_object_sink_v2_t) == 32, "v2 object sink size pinned"); +static_assert(PJ_PARSER_OBJECT_SINK_V2_MIN_SIZE == 32, "v2 object sink minimum size pinned"); +static_assert(offsetof(PJ_parser_functional_v2_t, struct_size) == 0, "functional v2 prefix pinned"); +static_assert(offsetof(PJ_parser_functional_v2_t, parse_scalars) == 8, "functional v2 scalar slot pinned"); +static_assert(offsetof(PJ_parser_functional_v2_t, parse_object) == 16, "functional v2 object slot pinned"); +static_assert(sizeof(PJ_parser_functional_v2_t) == 24, "functional v2 size pinned"); +static_assert(PJ_PARSER_FUNCTIONAL_V2_MIN_SIZE == 24, "functional v2 minimum size pinned"); + +static_assert(sizeof(PJ_route_classification_v1_t) == 8, "route classification v1 size pinned"); +static_assert(offsetof(PJ_route_classification_v1_t, route_flags) == 0, "route flags offset pinned"); +static_assert(offsetof(PJ_route_classification_v1_t, match) == 2, "route match offset pinned"); +static_assert(offsetof(PJ_route_classification_v1_t, status) == 4, "route status offset pinned"); +static_assert(offsetof(PJ_route_classification_v1_t, object_type) == 6, "route object type offset pinned"); +static_assert(offsetof(PJ_parser_route_claims_v1_t, struct_size) == 0, "route claims prefix pinned"); +static_assert(offsetof(PJ_parser_route_claims_v1_t, classify_routes) == 8, "route claims callback pinned"); +static_assert(sizeof(PJ_parser_route_claims_v1_t) == 16, "route claims extension size pinned"); +static_assert(PJ_PARSER_ROUTE_CLAIMS_V1_MIN_SIZE == 16, "route claims minimum size pinned"); #endif +static_assert(PJ_PARSER_MODULE_ABI_VERSION == 1, "parser module ABI version pinned"); +static_assert(PJ_MODULE_OK == 0, "parser module OK result pinned"); +static_assert(PJ_MODULE_DECLINE == 1, "parser module DECLINE result pinned"); +static_assert(PJ_MODULE_ERR_GENERIC == -1, "parser module generic error pinned"); +static_assert(PJ_MODULE_ERR_BAD_TOKEN == -2, "parser module bad-token error pinned"); +static_assert(PJ_MODULE_ERR_MALFORMED_INPUT == -3, "parser module malformed-input error pinned"); +static_assert(PJ_MODULE_ERR_BAD_CLAIM_INDEX == -4, "parser module bad-claim-index error pinned"); +static_assert(PJ_MODULE_ERR_ALLOCATION_FAILURE == -5, "parser module allocation error pinned"); +static_assert(PJ_MODULE_CREATION_ERROR_TOKEN == 0, "parser module creation-error token pinned"); +static_assert(PJ_MODULE_ERROR_BUFFER_SIZE == 512, "parser module error buffer size pinned"); +static_assert(std::string_view(PJ_MODULE_ABI_EXPORT_NAME) == "pj_module_abi", "parser module ABI export name pinned"); +static_assert( + std::string_view(PJ_MODULE_CREATE_EXPORT_NAME) == "pj_module_create", "parser module create export name pinned"); +static_assert( + std::string_view(PJ_MODULE_DESTROY_EXPORT_NAME) == "pj_module_destroy", "parser module destroy export name pinned"); +static_assert( + std::string_view(PJ_MODULE_BIND_EXPORT_NAME) == "pj_module_bind", "parser module bind export name pinned"); +static_assert( + std::string_view(PJ_MODULE_PARSE_EXPORT_NAME) == "pj_module_parse", "parser module parse export name pinned"); +static_assert( + std::string_view(PJ_MODULE_LAST_ERROR_EXPORT_NAME) == "pj_module_last_error", + "parser module last-error export name pinned"); +static_assert( + std::string_view(PJ_MODULE_ALLOC_EXPORT_NAME) == "pj_module_alloc", "parser module alloc export name pinned"); +static_assert( + std::string_view(PJ_MODULE_FREE_EXPORT_NAME) == "pj_module_free", "parser module free export name pinned"); +static_assert( + std::string_view(PJ_MODULE_MANIFEST_ADDR_EXPORT_NAME) == "pj_module_manifest_addr", + "native parser module manifest-address export name pinned"); +static_assert( + std::string_view(PJ_MODULE_MANIFEST_LEN_EXPORT_NAME) == "pj_module_manifest_len", + "native parser module manifest-length export name pinned"); +static_assert( + std::string_view(PJ_PARSER_MODULE_MANIFEST_SECTION_NAME) == "pj_parser_module_manifest", + "wasm parser module manifest section name pinned"); +static_assert(sizeof(PJ_module_parse_fn_t) == sizeof(void*), "parser module function pointer width pinned"); + static_assert(sizeof(PJ_payload_anchor_t) == 16, "PJ_payload_anchor_t pinned (ctx + release fn ptr)"); static_assert(offsetof(PJ_payload_anchor_t, ctx) == 0, "ctx at offset 0"); static_assert(offsetof(PJ_payload_anchor_t, release) == 8, "release at offset 8"); diff --git a/pj_base/tests/parser_module_abi_test.cpp b/pj_base/tests/parser_module_abi_test.cpp new file mode 100644 index 00000000..0855b6ef --- /dev/null +++ b/pj_base/tests/parser_module_abi_test.cpp @@ -0,0 +1,289 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_base/parser_module_abi.h" + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin_object_abi.h" + +namespace { + +PJ::Span bytesOf(std::string_view value) { + return {reinterpret_cast(value.data()), value.size()}; +} + +bool spansEqual(PJ::Span lhs, PJ::Span rhs) { + return lhs.size() == rhs.size() && std::equal(lhs.begin(), lhs.end(), rhs.begin()); +} + +const std::vector kBindingInfoGolden{ + 0x01, 0x00, 0x02, 0x00, // version, object route + 0x04, 0x03, 0x02, 0x01, // claim index + 0x03, 0x00, 0x00, 0x00, // expected object type, reserved + 0x06, 0x00, 0x00, 0x00, // field count + 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, // encoding + 0x43, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, // type name + 0x44, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, // schema + 0x46, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, // claim id + 0x47, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, // config JSON + 0x49, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // schema digest + 0x63, 0x64, 0x72, 0x54, 0x00, 0xFF, 0x63, 0x7B, 0x7D, +}; + +const std::vector kParseInputGolden{ + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // flags, padding + 0xFE, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // timestamp -2 + 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // payload length + 0xAA, 0x00, 0xFF, +}; + +const std::vector kObjectOutputGolden{ + 0x01, 0x00, 0x02, 0x00, // version, object route, reserved + 0x03, 0x00, 0x01, 0x00, // object type, splice count + 0x09, 0x00, 0x00, 0x00, // splice field number + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // splice offset + 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // splice length + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // wire length + 0x08, 0x01, +}; + +const std::vector kScalarOutputGolden{ + 0x01, 0x00, 0x01, 0x00, // version, scalar route, reserved + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // timestamp flag, padding + 0xFE, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // timestamp -2 + 0x05, 0x00, 0x00, 0x00, // field count + 0x64, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, // name a, f64 + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF8, 0x3F, // 1.5 + 0x65, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, // name b, i64 + 0xFE, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // -2 + 0x66, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x02, // name c, u64 + 0x08, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x67, 0x00, + 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x03, // name d, bool + 0x01, 0x68, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04, // name e, string + 0x02, 0x00, 0x00, 0x00, 0x68, 0x69, // "hi" + 0x61, 0x62, 0x63, 0x64, 0x65, // field names +}; + +TEST(ParserModuleAbi, BindingInfoV1MatchesGoldenAndRoundTrips) { + const std::array schema{0x00, 0xFF}; + const PJ::parser_module::BindingInfoV1 input{ + .route = PJ::parser_module::Route::kObject, + .claim_index = 0x01020304, + .expected_object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .encoding = bytesOf("cdr"), + .type_name = bytesOf("T"), + .schema = schema, + .claim_id = bytesOf("c"), + .config_json = bytesOf("{}"), + .schema_digest = {}, + }; + + const auto written = PJ::parser_module::writeBindingInfoV1(input); + ASSERT_TRUE(written) << written.error(); + EXPECT_EQ(*written, kBindingInfoGolden); + + const auto read = PJ::parser_module::readBindingInfoV1(kBindingInfoGolden); + ASSERT_TRUE(read) << read.error(); + EXPECT_EQ(read->route, PJ::parser_module::Route::kObject); + EXPECT_EQ(read->claim_index, 0x01020304U); + EXPECT_EQ(read->expected_object_type, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + EXPECT_TRUE(spansEqual(bytesOf("cdr"), read->encoding)); + EXPECT_TRUE(spansEqual(bytesOf("T"), read->type_name)); + EXPECT_TRUE(spansEqual(PJ::Span(schema), read->schema)); + EXPECT_TRUE(spansEqual(bytesOf("c"), read->claim_id)); + EXPECT_TRUE(spansEqual(bytesOf("{}"), read->config_json)); + EXPECT_TRUE(read->schema_digest.empty()); +} + +TEST(ParserModuleAbi, BindingInfoReaderIgnoresValidatedTrailingFields) { + std::vector with_extra = kBindingInfoGolden; + with_extra[12] = 7; + for (size_t index = 0; index < 6; ++index) { + with_extra[16 + index * 8] = static_cast(with_extra[16 + index * 8] + 8); + } + with_extra.insert(with_extra.begin() + 64, {0x51, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00}); + with_extra.push_back(0x78); + + const auto read = PJ::parser_module::readBindingInfoV1(with_extra); + ASSERT_TRUE(read) << read.error(); + EXPECT_TRUE(spansEqual(bytesOf("cdr"), read->encoding)); + EXPECT_TRUE(spansEqual(bytesOf("{}"), read->config_json)); +} + +TEST(ParserModuleAbi, BindingInfoReaderRejectsTruncationAndMalformedRanges) { + for (size_t size = 0; size < kBindingInfoGolden.size(); ++size) { + EXPECT_FALSE(PJ::parser_module::readBindingInfoV1({kBindingInfoGolden.data(), size})) << "prefix size " << size; + } + + auto malformed = kBindingInfoGolden; + malformed[16] = 0xFF; + EXPECT_FALSE(PJ::parser_module::readBindingInfoV1(malformed)); + malformed = kBindingInfoGolden; + malformed[12] = 5; + EXPECT_FALSE(PJ::parser_module::readBindingInfoV1(malformed)); + malformed = kBindingInfoGolden; + malformed[0] = 2; + EXPECT_FALSE(PJ::parser_module::readBindingInfoV1(malformed)); +} + +TEST(ParserModuleAbi, ParseInputV1MatchesGoldenAndRoundTrips) { + const std::array payload{0xAA, 0x00, 0xFF}; + const auto written = PJ::parser_module::writeParseInputV1( + PJ::parser_module::ParseInputV1{.has_timestamp = true, .timestamp_ns = -2, .payload = payload}); + ASSERT_TRUE(written) << written.error(); + EXPECT_EQ(*written, kParseInputGolden); + + const auto read = PJ::parser_module::readParseInputV1(kParseInputGolden); + ASSERT_TRUE(read) << read.error(); + EXPECT_TRUE(read->has_timestamp); + EXPECT_EQ(read->timestamp_ns, -2); + EXPECT_TRUE(spansEqual(PJ::Span(payload), read->payload)); +} + +TEST(ParserModuleAbi, ParseInputReaderRejectsTruncationFlagsAndLengthMismatch) { + for (size_t size = 0; size < kParseInputGolden.size(); ++size) { + EXPECT_FALSE(PJ::parser_module::readParseInputV1({kParseInputGolden.data(), size})) << "prefix size " << size; + } + auto malformed = kParseInputGolden; + malformed[0] = 2; + EXPECT_FALSE(PJ::parser_module::readParseInputV1(malformed)); + malformed = kParseInputGolden; + malformed[16] = 2; + EXPECT_FALSE(PJ::parser_module::readParseInputV1(malformed)); + malformed.push_back(0); + EXPECT_FALSE(PJ::parser_module::readParseInputV1(malformed)); +} + +TEST(ParserModuleAbi, ObjectOutputV1MatchesGoldenAndRoundTrips) { + const std::array wire{0x08, 0x01}; + const PJ::parser_module::OutputDescriptorV1 output = PJ::parser_module::ObjectOutputV1{ + .object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .splice = PJ::parser_module::ObjectSpliceV1{.field_number = 9, .input_offset = 2, .input_length = 3}, + .wire = wire, + }; + const auto written = PJ::parser_module::writeOutputDescriptorV1(output); + ASSERT_TRUE(written) << written.error(); + EXPECT_EQ(*written, kObjectOutputGolden); + + const auto read = PJ::parser_module::readOutputDescriptorV1(kObjectOutputGolden); + ASSERT_TRUE(read) << read.error(); + const auto* object = std::get_if(&*read); + ASSERT_NE(object, nullptr); + EXPECT_EQ(object->object_type, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + ASSERT_TRUE(object->splice.has_value()); + EXPECT_EQ(object->splice->field_number, 9U); + EXPECT_EQ(object->splice->input_offset, 2U); + EXPECT_EQ(object->splice->input_length, 3U); + EXPECT_TRUE(spansEqual(PJ::Span(wire), object->wire)); +} + +TEST(ParserModuleAbi, ScalarOutputV1MatchesGoldenAndRoundTripsEveryValueKind) { + PJ::parser_module::ScalarOutputV1 scalar{ + .has_timestamp = true, + .timestamp_ns = -2, + .fields = {}, + }; + scalar.fields = { + {.name = "a", .value = 1.5}, + {.name = "b", .value = int64_t{-2}}, + {.name = "c", .value = UINT64_C(0x0102030405060708)}, + {.name = "d", .value = true}, + {.name = "e", .value = std::string_view("hi")}, + }; + const PJ::parser_module::OutputDescriptorV1 output = scalar; + const auto written = PJ::parser_module::writeOutputDescriptorV1(output); + ASSERT_TRUE(written) << written.error(); + EXPECT_EQ(*written, kScalarOutputGolden); + + const auto read = PJ::parser_module::readOutputDescriptorV1(kScalarOutputGolden); + ASSERT_TRUE(read) << read.error(); + const auto* decoded = std::get_if(&*read); + ASSERT_NE(decoded, nullptr); + EXPECT_TRUE(decoded->has_timestamp); + EXPECT_EQ(decoded->timestamp_ns, -2); + ASSERT_EQ(decoded->fields.size(), 5U); + EXPECT_EQ(decoded->fields[0].name, "a"); + EXPECT_DOUBLE_EQ(std::get(decoded->fields[0].value), 1.5); + EXPECT_EQ(std::get(decoded->fields[1].value), -2); + EXPECT_EQ(std::get(decoded->fields[2].value), UINT64_C(0x0102030405060708)); + EXPECT_TRUE(std::get(decoded->fields[3].value)); + EXPECT_EQ(std::get(decoded->fields[4].value), "hi"); +} + +TEST(ParserModuleAbi, OutputReaderRejectsTruncationAndMalformedDescriptors) { + for (size_t size = 0; size < kObjectOutputGolden.size(); ++size) { + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1({kObjectOutputGolden.data(), size})) + << "object prefix size " << size; + } + for (size_t size = 0; size < kScalarOutputGolden.size(); ++size) { + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1({kScalarOutputGolden.data(), size})) + << "scalar prefix size " << size; + } + + auto malformed = kObjectOutputGolden; + malformed[6] = 2; + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1(malformed)); + malformed = kObjectOutputGolden; + malformed[2] = 3; + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1(malformed)); + auto malformed_scalar = kScalarOutputGolden; + malformed_scalar[32] = 9; + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1(malformed_scalar)); + malformed_scalar = kScalarOutputGolden; + malformed_scalar[24] = 0xFF; + EXPECT_FALSE(PJ::parser_module::readOutputDescriptorV1(malformed_scalar)); +} + +TEST(BuiltinObjectSpliceTable, ContainsOnlyUnambiguousTopLevelBulkByteFields) { + const std::array expected{ + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_IMAGE, 0, 7}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, 0, 9}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_DEPTH_IMAGE, 0, 5}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_OCCUPANCY_GRID, 0, 7}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_COMPRESSED_POINTCLOUD, 0, 4}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_MESH3D, 0, 7}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_VIDEO_FRAME, 0, 3}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_OCCUPANCY_GRID_UPDATE, 0, 7}, + PJ_builtin_object_splice_field_v1_t{PJ_BUILTIN_OBJECT_TYPE_VOXEL_GRID, 0, 12}, + }; + uint32_t count = 0; + const auto* table = pj_builtin_object_splice_fields_v1(&count); + ASSERT_NE(table, nullptr); + ASSERT_EQ(count, expected.size()); + for (size_t index = 0; index < expected.size(); ++index) { + EXPECT_EQ(table[index].object_type, expected[index].object_type); + EXPECT_EQ(table[index].reserved, 0); + EXPECT_EQ(table[index].field_number, expected[index].field_number); + uint32_t field_number = 0; + EXPECT_TRUE(pj_builtin_object_splice_field_number_v1(expected[index].object_type, &field_number)); + EXPECT_EQ(field_number, expected[index].field_number); + } + + for (const uint16_t absent : { + uint16_t{PJ_BUILTIN_OBJECT_TYPE_NONE}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_IMAGE_ANNOTATIONS}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_FRAME_TRANSFORMS}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_SCENE_ENTITIES}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_ROBOT_DESCRIPTION}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_CAMERA_INFO}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_LOG}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_POSES_IN_FRAME}, + uint16_t{PJ_BUILTIN_OBJECT_TYPE_PLOT_MARKERS}, + }) { + uint32_t field_number = 99; + EXPECT_FALSE(pj_builtin_object_splice_field_number_v1(absent, &field_number)); + EXPECT_EQ(field_number, 99U); + } + EXPECT_FALSE(pj_builtin_object_splice_field_number_v1(PJ_BUILTIN_OBJECT_TYPE_IMAGE, nullptr)); +} + +} // namespace diff --git a/pj_base/tests/parser_module_kit_test.cpp b/pj_base/tests/parser_module_kit_test.cpp new file mode 100644 index 00000000..1d6c6108 --- /dev/null +++ b/pj_base/tests/parser_module_kit_test.cpp @@ -0,0 +1,494 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module/module.hpp" + +namespace { + +void alignCdr(std::vector& bytes, size_t alignment) { + const size_t relative = bytes.size() - 4; + const size_t padding = (alignment - (relative % alignment)) % alignment; + bytes.insert(bytes.end(), padding, 0); +} + +void cdrU16(std::vector& bytes, uint16_t value, bool little = true) { + alignCdr(bytes, 2); + if (little) { + bytes.push_back(static_cast(value)); + bytes.push_back(static_cast(value >> 8U)); + } else { + bytes.push_back(static_cast(value >> 8U)); + bytes.push_back(static_cast(value)); + } +} + +void cdrU32(std::vector& bytes, uint32_t value, bool little = true) { + alignCdr(bytes, 4); + for (size_t index = 0; index < 4; ++index) { + const size_t shift = little ? index * 8U : (3 - index) * 8U; + bytes.push_back(static_cast(value >> shift)); + } +} + +void cdrString(std::vector& bytes, std::string_view value) { + cdrU32(bytes, static_cast(value.size() + 1)); + bytes.insert(bytes.end(), value.begin(), value.end()); + bytes.push_back(0); +} + +std::vector cdrFixture() { + std::vector bytes{0, 1, 0, 0}; + cdrU32(bytes, 99); + cdrU32(bytes, 7); + cdrU32(bytes, 8); + cdrString(bytes, "map"); + cdrU32(bytes, 2); + cdrU32(bytes, 3); + bytes.insert(bytes.end(), {10, 20, 30}); + cdrU16(bytes, 42); + return bytes; +} + +constexpr std::string_view kCdrSchema = R"(uint32 prefix +std_msgs/Header header +uint32 width +uint8[] payload +uint16 tail +================================================================================ +MSG: std_msgs/Header +builtin_interfaces/Time stamp +string frame_id +================================================================================ +MSG: builtin_interfaces/Time +int32 sec +uint32 nanosec +)"; + +TEST(ParserModuleCdrReader, ReadsEndianAwarePrimitivesArraysSequencesAndStrings) { + std::vector little{0, 1, 0, 0}; + cdrU16(little, 0x1234); + cdrU16(little, 0x5678); + cdrU32(little, 3); + little.insert(little.end(), {1, 2, 3}); + cdrString(little, "ok"); + + pj::CdrReader reader({little.data(), little.size()}); + std::array fixed{}; + ASSERT_TRUE(reader.readFixedArray(fixed).isOk()); + EXPECT_EQ(fixed, (std::array{0x1234, 0x5678})); + auto sequence = reader.readByteSequence(); + ASSERT_TRUE(sequence.hasValue()) << sequence.status().message(); + EXPECT_EQ(std::vector(sequence->data, sequence->data + sequence->size), (std::vector{1, 2, 3})); + auto text = reader.readString(); + ASSERT_TRUE(text.hasValue()) << text.status().message(); + EXPECT_EQ(*text, "ok"); + + std::vector big{0, 0, 0, 0}; + cdrU32(big, 0x01020304, false); + pj::CdrReader big_reader({big.data(), big.size()}); + auto value = big_reader.readU32(); + ASSERT_TRUE(value.hasValue()) << value.status().message(); + EXPECT_EQ(*value, 0x01020304U); +} + +TEST(ParserModuleCdrReader, RejectsMalformedLengthsTruncationAndDepthOverflow) { + for (const std::vector& bytes : { + std::vector{}, + std::vector{0, 1, 0}, + std::vector{0, 6, 0, 0}, + std::vector{0, 1, 0, 0, 4, 0, 0, 0, 'x'}, + }) { + pj::CdrReader reader({bytes.data(), bytes.size()}); + if (bytes.size() == 9) { + EXPECT_FALSE(reader.readString().hasValue()); + } else { + EXPECT_FALSE(reader.status().isOk()); + } + } + + std::vector bytes{0, 1, 0, 0}; + cdrU32(bytes, 100); + pj::CdrReader sequence_reader({bytes.data(), bytes.size()}); + EXPECT_FALSE(sequence_reader.readSequenceLength(4).hasValue()); + + pj::CdrReader depth_reader({bytes.data(), bytes.size()}); + for (size_t depth = 0; depth < pj::CdrReader::kMaxTraversalDepth; ++depth) { + ASSERT_TRUE(depth_reader.enterStruct().isOk()); + } + EXPECT_FALSE(depth_reader.enterStruct().isOk()); +} + +TEST(ParserModuleCore, BumpArenaReportsAlignmentErrorsAndReusesStorage) { + pj::BumpArena arena; + auto empty = arena.allocate(0); + ASSERT_TRUE(empty.hasValue()) << empty.status().message(); + EXPECT_EQ(empty->data, nullptr); + EXPECT_EQ(arena.used(), 0U); + EXPECT_FALSE(arena.allocate(4, 3).hasValue()); + EXPECT_FALSE(arena.allocate(4, alignof(std::max_align_t) * 2).hasValue()); + + auto first = arena.allocate(3, 1); + auto second = arena.allocate(4, 4); + ASSERT_TRUE(first.hasValue()) << first.status().message(); + ASSERT_TRUE(second.hasValue()) << second.status().message(); + EXPECT_EQ(reinterpret_cast(second->data) % 4U, 0U); + EXPECT_EQ(arena.used(), 8U); + arena.reset(); + EXPECT_EQ(arena.used(), 0U); + EXPECT_TRUE(arena.allocate(8, 8).hasValue()); +} + +TEST(ParserModuleCdrFieldLocator, CompilesNestedPlanAndCachesOneMessageTraversal) { + pj::CdrFieldLocator locator(kCdrSchema); + ASSERT_TRUE(locator.status().isOk()) << locator.status().message(); + auto plan = + locator.locate({"header.stamp.sec", "header.stamp.nanosec", "header.frame_id", "width", "payload", "tail"}); + ASSERT_TRUE(plan.hasValue()) << plan.status().message(); + + const auto sec = plan->field("header.stamp.sec"); + const auto nanos = plan->field("header.stamp.nanosec"); + const auto frame = plan->field("header.frame_id"); + const auto width = plan->field("width"); + const auto payload = plan->field("payload"); + const auto tail = plan->field("tail"); + ASSERT_TRUE(sec && nanos && frame && width && payload && tail); + + const auto bytes = cdrFixture(); + pj::CdrReader reader({bytes.data(), bytes.size()}, *plan); + auto last = reader.u16(*tail); + ASSERT_TRUE(last.hasValue()) << last.status().message(); + EXPECT_EQ(*last, 42U); + EXPECT_EQ(reader.traversalCount(), 1U); + EXPECT_EQ(*reader.u32(*width), 2U); + EXPECT_EQ(*reader.string(*frame), "map"); + EXPECT_EQ(*reader.i32(*sec), 7); + EXPECT_EQ(*reader.u32(*nanos), 8U); + auto data = reader.bytes(*payload); + ASSERT_TRUE(data.hasValue()); + EXPECT_EQ(std::vector(data->data, data->data + data->size), (std::vector{10, 20, 30})); + auto reference = reader.spanRef(*payload); + ASSERT_TRUE(reference.hasValue()); + EXPECT_EQ(reference->offset, static_cast(data->data - bytes.data())); + EXPECT_EQ(reference->length, 3U); + EXPECT_EQ(reader.traversalCount(), 1U); +} + +TEST(ParserModuleCdrFieldLocator, AcceptsBoundedAndStringContainersAndRejectsMissingFieldsTruncationAndDeepPaths) { + pj::CdrFieldLocator bounded("uint8[<=8] data\n"); + EXPECT_TRUE(bounded.status().isOk()) << bounded.status().message(); + + pj::CdrFieldLocator valid(kCdrSchema); + EXPECT_FALSE(valid.locate({"missing"}).hasValue()); + auto plan = valid.locate({"payload"}); + ASSERT_TRUE(plan.hasValue()); + auto bytes = cdrFixture(); + bytes.pop_back(); + pj::CdrReader truncated({bytes.data(), bytes.size()}, *plan); + EXPECT_FALSE(truncated.bytes(*plan->field("payload")).hasValue()); + + std::string schema = "T1 next\n"; + std::string path = "next"; + for (size_t depth = 1; depth <= 64; ++depth) { + schema += "MSG: T" + std::to_string(depth) + "\n"; + if (depth < 64) { + schema += "T" + std::to_string(depth + 1) + " next\n"; + path += ".next"; + } else { + schema += "uint32 value\n"; + path += ".value"; + } + } + pj::CdrFieldLocator deep(schema); + EXPECT_FALSE(deep.status().isOk()); +} + +TEST(ParserModuleCdrFieldLocator, TraversesBoundedStringsAndArraysOrSequencesOfStrings) { + constexpr std::string_view schema = + "uint8[<=4] data\nstring<=4 label\nstring[2] names\nstring[] aliases\nuint16 tail\n"; + pj::CdrFieldLocator locator(schema); + ASSERT_TRUE(locator.status().isOk()) << locator.status().message(); + auto plan = locator.locate({"data", "label", "tail"}); + ASSERT_TRUE(plan.hasValue()) << plan.status().message(); + + std::vector bytes{0, 1, 0, 0}; + cdrU32(bytes, 3); + bytes.insert(bytes.end(), {1, 2, 3}); + cdrString(bytes, "tag"); + cdrString(bytes, "one"); + cdrString(bytes, "two"); + cdrU32(bytes, 2); + cdrString(bytes, "a"); + cdrString(bytes, "b"); + cdrU16(bytes, 77); + + pj::CdrReader reader({bytes.data(), bytes.size()}, *plan); + EXPECT_EQ(*reader.string(*plan->field("label")), "tag"); + EXPECT_EQ(*reader.u16(*plan->field("tail")), 77U); + auto data = reader.bytes(*plan->field("data")); + ASSERT_TRUE(data.hasValue()); + EXPECT_EQ(data->size, 3U); + + auto too_many = bytes; + too_many[4] = 5; + pj::CdrReader invalid({too_many.data(), too_many.size()}, *plan); + EXPECT_FALSE(invalid.bytes(*plan->field("data")).hasValue()); +} + +TEST(ParserModuleCdrFieldLocator, RejectsCyclicSchemasAtBindAndZeroConsumptionSequencesAtParse) { + pj::CdrFieldLocator cyclic( + "pkg/Node[] children\n================================================================================\n" + "MSG: pkg/Node\npkg/Node[] children\n"); + EXPECT_FALSE(cyclic.status().isOk()); + EXPECT_NE(cyclic.status().message().find("cyclic"), std::string_view::npos); + + pj::CdrFieldLocator empty_sequence( + "pkg/Empty[] entries\nuint16 " + "tail\n================================================================================\n" + "MSG: pkg/Empty\nuint32 CONSTANT=1\n"); + ASSERT_TRUE(empty_sequence.status().isOk()) << empty_sequence.status().message(); + auto plan = empty_sequence.locate({"tail"}); + ASSERT_TRUE(plan.hasValue()) << plan.status().message(); + std::vector bytes{0, 1, 0, 0}; + cdrU32(bytes, 1); + cdrU16(bytes, 9); + pj::CdrReader reader({bytes.data(), bytes.size()}, *plan); + EXPECT_FALSE(reader.u16(*plan->field("tail")).hasValue()); + EXPECT_NE(reader.status().message().find("zero serialized minimum"), std::string_view::npos); +} + +TEST(ParserModuleCdrFieldLocator, PlannedTraversalRewindsAfterStreamingReads) { + pj::CdrFieldLocator locator(kCdrSchema); + auto plan = locator.locate({"tail"}); + ASSERT_TRUE(plan.hasValue()) << plan.status().message(); + const auto bytes = cdrFixture(); + pj::CdrReader reader({bytes.data(), bytes.size()}, *plan); + ASSERT_TRUE(reader.readU32().hasValue()); + auto tail = reader.u16(*plan->field("tail")); + ASSERT_TRUE(tail.hasValue()) << tail.status().message(); + EXPECT_EQ(*tail, 42U); + EXPECT_EQ(reader.traversalCount(), 1U); +} + +pj::Blob makeMessage(std::initializer_list> fields) { + pj::WireWriter writer; + for (const auto& field : fields) { + EXPECT_TRUE(writer.varintField(field.first, field.second).isOk()); + } + return writer.take(); +} + +pj::Blob descriptorField( + std::string_view name, uint32_t number, uint32_t label, uint32_t type, std::string_view type_name = {}) { + pj::WireWriter writer; + EXPECT_TRUE(writer.stringField(1, name).isOk()); + EXPECT_TRUE(writer.varintField(3, number).isOk()); + EXPECT_TRUE(writer.varintField(4, label).isOk()); + EXPECT_TRUE(writer.varintField(5, type).isOk()); + if (!type_name.empty()) { + EXPECT_TRUE(writer.stringField(6, type_name).isOk()); + } + return writer.take(); +} + +pj::Blob descriptorSetFixture() { + const auto child_count = descriptorField("count", 3, 1, 5); + pj::WireWriter child; + EXPECT_TRUE(child.stringField(1, "Child").isOk()); + EXPECT_TRUE(child.lengthDelimited(2, child_count.view()).isOk()); + + const auto root_child = descriptorField("child", 1, 1, 11, ".example.Child"); + const auto root_value = descriptorField("value", 2, 1, 13); + pj::WireWriter root; + EXPECT_TRUE(root.stringField(1, "Root").isOk()); + EXPECT_TRUE(root.lengthDelimited(2, root_child.view()).isOk()); + EXPECT_TRUE(root.lengthDelimited(2, root_value.view()).isOk()); + + pj::WireWriter file; + EXPECT_TRUE(file.stringField(2, "example").isOk()); + EXPECT_TRUE(file.messageField(4, root).isOk()); + EXPECT_TRUE(file.messageField(4, child).isOk()); + pj::WireWriter set; + EXPECT_TRUE(set.messageField(1, file).isOk()); + return set.take(); +} + +pj::Blob nestedDescriptor(size_t depth, size_t nested_count, const std::string& parent) { + const std::string name = depth == 0 ? "Root" : "N" + std::to_string(depth); + const std::string full_name = parent.empty() ? name : parent + "." + name; + pj::WireWriter message; + EXPECT_TRUE(message.stringField(1, name).isOk()); + if (depth == nested_count) { + const auto value = descriptorField("value", 1, 1, 13); + EXPECT_TRUE(message.lengthDelimited(2, value.view()).isOk()); + } else { + const std::string child_name = "N" + std::to_string(depth + 1); + const auto next = descriptorField("next", 1, 1, 11, "." + full_name + "." + child_name); + const auto child = nestedDescriptor(depth + 1, nested_count, full_name); + EXPECT_TRUE(message.lengthDelimited(2, next.view()).isOk()); + EXPECT_TRUE(message.lengthDelimited(3, child.view()).isOk()); + } + return message.take(); +} + +pj::Blob descriptorSetWithRoot(pj::Blob root) { + pj::WireWriter actual_file; + EXPECT_TRUE(actual_file.lengthDelimited(4, root.view()).isOk()); + pj::WireWriter set; + EXPECT_TRUE(set.messageField(1, actual_file).isOk()); + return set.take(); +} + +TEST(ParserModuleProtoReader, HandlesUnknownPackedUnpackedLastWinsAndMalformedInput) { + pj::WireWriter writer; + ASSERT_TRUE(writer.varintField(9, 123).isOk()); + ASSERT_TRUE(writer.varintField(1, 10).isOk()); + ASSERT_TRUE(writer.varintField(2, 1).isOk()); + pj::WireWriter packed; + ASSERT_TRUE(packed.varint(2).isOk()); + ASSERT_TRUE(packed.varint(300).isOk()); + ASSERT_TRUE(writer.lengthDelimited(2, packed.view()).isOk()); + ASSERT_TRUE(writer.varintField(1, 20).isOk()); + + const auto bytes = writer.take(); + pj::ProtoReader reader(bytes.view()); + auto scalar = reader.varint(1); + ASSERT_TRUE(scalar.hasValue()) << scalar.status().message(); + EXPECT_EQ(*scalar, 20U); + auto repeated = reader.repeatedVarints(2); + ASSERT_TRUE(repeated.hasValue()) << repeated.status().message(); + EXPECT_EQ(*repeated, (std::vector{1, 2, 300})); + + const std::array truncated_varint{0x08, 0x80}; + EXPECT_FALSE(pj::ProtoReader({truncated_varint.data(), truncated_varint.size()}).varint(1).hasValue()); + const std::array truncated_group{0x0B, 0x10}; + EXPECT_FALSE(pj::ProtoReader({truncated_group.data(), truncated_group.size()}).matching(2).hasValue()); + EXPECT_FALSE(pj::ProtoReader({nullptr, 1}).matching(1).hasValue()); + EXPECT_FALSE(pj::ProtoReader(bytes.view()).matching(UINT32_C(0x20000000)).hasValue()); +} + +TEST(ParserModuleProtoReader, BoundsMatchingResultsWithoutThrowing) { + std::vector bytes; + bytes.reserve((pj::ProtoReader::FieldList::kMaximumFields + 1) * 2); + for (size_t index = 0; index <= pj::ProtoReader::FieldList::kMaximumFields; ++index) { + bytes.push_back(0x08); + bytes.push_back(0x00); + } + auto fields = pj::ProtoReader({bytes.data(), bytes.size()}).matching(1); + ASSERT_FALSE(fields.hasValue()); + EXPECT_NE(fields.status().message().find("configured limit"), std::string_view::npos); +} + +TEST(ParserModuleProtoReader, ReadsZigzagAndPackedOrUnpackedFixedValues) { + pj::WireWriter writer; + ASSERT_TRUE(writer.varintField(1, 3).isOk()); + ASSERT_TRUE(writer.fixed32Field(2, UINT32_C(0x01020304)).isOk()); + const std::array packed32{0x08, 0x07, 0x06, 0x05, 0x0C, 0x0B, 0x0A, 0x09}; + ASSERT_TRUE(writer.lengthDelimited(2, {packed32.data(), packed32.size()}).isOk()); + ASSERT_TRUE(writer.fixed64Field(3, UINT64_C(0x0102030405060708)).isOk()); + const std::array packed64{0x18, 0x17, 0x16, 0x15, 0x14, 0x13, 0x12, 0x11}; + ASSERT_TRUE(writer.lengthDelimited(3, {packed64.data(), packed64.size()}).isOk()); + + const auto bytes = writer.take(); + pj::ProtoReader reader(bytes.view()); + EXPECT_EQ(*reader.zigzag(1), -2); + auto fixed32 = reader.repeatedFixed32(2); + ASSERT_TRUE(fixed32.hasValue()) << fixed32.status().message(); + EXPECT_EQ(*fixed32, (std::vector{UINT32_C(0x01020304), UINT32_C(0x05060708), UINT32_C(0x090A0B0C)})); + auto fixed64 = reader.repeatedFixed64(3); + ASSERT_TRUE(fixed64.hasValue()) << fixed64.status().message(); + EXPECT_EQ(*fixed64, (std::vector{UINT64_C(0x0102030405060708), UINT64_C(0x1112131415161718)})); +} + +TEST(ParserModuleProtoReader, EnforcesSubmessageDepthCap) { + pj::Blob nested = makeMessage({{2, 1}}); + for (size_t depth = 0; depth < 65; ++depth) { + pj::WireWriter wrapper; + ASSERT_TRUE(wrapper.lengthDelimited(1, nested.view()).isOk()); + nested = wrapper.take(); + } + pj::ProtoReader reader(nested.view()); + for (size_t depth = 0; depth < pj::ProtoReader::kMaxRecursionDepth; ++depth) { + auto child = reader.submessage(1); + ASSERT_TRUE(child.hasValue()) << depth << ": " << child.status().message(); + reader = *child; + } + EXPECT_FALSE(reader.submessage(1).hasValue()); +} + +TEST(ParserModuleProtoFieldLocator, CompilesFileDescriptorSetIntoFieldNumberPaths) { + const auto descriptors = descriptorSetFixture(); + pj::ProtoFieldLocator locator(descriptors.view(), ".example.Root"); + ASSERT_TRUE(locator.status().isOk()) << locator.status().message(); + auto plan = locator.locate({"child.count", "value"}); + ASSERT_TRUE(plan.hasValue()) << plan.status().message(); + const auto child_count = plan->field("child.count"); + const auto value = plan->field("value"); + ASSERT_TRUE(child_count && value); + EXPECT_EQ(*plan->numberPath(*child_count), (std::vector{1, 3})); + EXPECT_EQ(*plan->numberPath(*value), (std::vector{2})); + + const auto child = makeMessage({{3, 17}}); + pj::WireWriter root; + ASSERT_TRUE(root.lengthDelimited(1, child.view()).isOk()); + ASSERT_TRUE(root.varintField(2, 8).isOk()); + const auto root_bytes = root.take(); + pj::ProtoReader reader(root_bytes.view()); + EXPECT_EQ(plan->locate(reader, *child_count)->integer, 17U); + EXPECT_EQ(plan->locate(reader, *value)->integer, 8U); + EXPECT_FALSE(locator.locate({"missing"}).hasValue()); +} + +TEST(ParserModuleProtoFieldLocator, RejectsMalformedLabelsTruncationAndDescriptorDepthOverflow) { + const auto invalid_field = descriptorField("value", 1, 4, 13); + pj::WireWriter invalid_root; + ASSERT_TRUE(invalid_root.stringField(1, "Root").isOk()); + ASSERT_TRUE(invalid_root.lengthDelimited(2, invalid_field.view()).isOk()); + const auto invalid_set = descriptorSetWithRoot(invalid_root.take()); + pj::ProtoFieldLocator invalid_label(invalid_set.view(), "Root"); + EXPECT_FALSE(invalid_label.status().isOk()); + + auto valid_set = descriptorSetFixture(); + pj::ProtoFieldLocator truncated({valid_set.data(), valid_set.size() - 1}, ".example.Root"); + EXPECT_FALSE(truncated.status().isOk()); + + auto deep_set = descriptorSetWithRoot(nestedDescriptor(0, 64, "")); + pj::ProtoFieldLocator too_deep(deep_set.view(), "Root"); + EXPECT_FALSE(too_deep.status().isOk()); + + const auto first = descriptorField("first", 1, 1, 13); + const auto duplicate_name = descriptorField("first", 2, 1, 13); + pj::WireWriter duplicate_name_root; + ASSERT_TRUE(duplicate_name_root.stringField(1, "Root").isOk()); + ASSERT_TRUE(duplicate_name_root.lengthDelimited(2, first.view()).isOk()); + ASSERT_TRUE(duplicate_name_root.lengthDelimited(2, duplicate_name.view()).isOk()); + const auto duplicate_name_set = descriptorSetWithRoot(duplicate_name_root.take()); + EXPECT_FALSE(pj::ProtoFieldLocator(duplicate_name_set.view(), "Root").status().isOk()); + + const auto duplicate_number = descriptorField("second", 1, 1, 13); + pj::WireWriter duplicate_number_root; + ASSERT_TRUE(duplicate_number_root.stringField(1, "Root").isOk()); + ASSERT_TRUE(duplicate_number_root.lengthDelimited(2, first.view()).isOk()); + ASSERT_TRUE(duplicate_number_root.lengthDelimited(2, duplicate_number.view()).isOk()); + const auto duplicate_number_set = descriptorSetWithRoot(duplicate_number_root.take()); + EXPECT_FALSE(pj::ProtoFieldLocator(duplicate_number_set.view(), "Root").status().isOk()); +} + +TEST(ParserModuleTime, RejectsInvalidAndOverflowingTimestamps) { + EXPECT_EQ(*pj::readRosTime(2, 3), INT64_C(2000000003)); + EXPECT_FALSE(pj::readRosTime(0, UINT32_C(1000000000)).hasValue()); + EXPECT_EQ(*pj::readProtoTimestamp(-1, 500000000), INT64_C(-500000000)); + EXPECT_FALSE(pj::readProtoTimestamp(std::numeric_limits::max(), 0).hasValue()); + EXPECT_FALSE(pj::readProtoTimestamp(0, -1).hasValue()); +} + +} // namespace diff --git a/pj_base/tests/parser_module_manifest_test.cpp b/pj_base/tests/parser_module_manifest_test.cpp new file mode 100644 index 00000000..ed88ac11 --- /dev/null +++ b/pj_base/tests/parser_module_manifest_test.cpp @@ -0,0 +1,73 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_base/parser_module_manifest.hpp" + +#include + +#include +#include +#include +#include +#include +#include + +namespace PJ::parser_module { +namespace { + +constexpr std::array kMinimalWasm{0x00, 0x61, 0x73, 0x6D, 0x01, 0x00, 0x00, 0x00}; +constexpr std::array kManifest{'{', '"', 'x', '"', ':', '1', '}'}; + +const std::vector kEmbeddedGolden{ + 0x00, 0x61, 0x73, 0x6D, 0x01, 0x00, 0x00, 0x00, // wasm preamble + 0x00, 0x21, // custom section, 33-byte payload + 0x19, // 25-byte custom-section name + 'p', 'j', '_', 'p', 'a', 'r', 's', 'e', 'r', '_', 'm', 'o', 'd', 'u', 'l', 'e', + '_', 'm', 'a', 'n', 'i', 'f', 'e', 's', 't', '{', '"', 'x', '"', ':', '1', '}', +}; + +TEST(ParserModuleManifest, AppendsGoldenSectionAndReadsExactBytes) { + auto embedded = appendManifestSection(kMinimalWasm, kManifest); + ASSERT_TRUE(embedded.has_value()) << embedded.error(); + EXPECT_EQ(*embedded, kEmbeddedGolden); + + auto decoded = readManifestSection(*embedded); + ASSERT_TRUE(decoded.has_value()) << decoded.error(); + EXPECT_TRUE(std::equal(decoded->begin(), decoded->end(), kManifest.begin(), kManifest.end())); +} + +TEST(ParserModuleManifest, RejectsZeroOrMultipleManifestSections) { + auto missing = readManifestSection(kMinimalWasm); + ASSERT_FALSE(missing.has_value()); + EXPECT_NE(missing.error().find("no parser-module manifest"), std::string::npos); + + std::vector duplicate = kEmbeddedGolden; + duplicate.insert( + duplicate.end(), kEmbeddedGolden.begin() + static_cast(kMinimalWasm.size()), + kEmbeddedGolden.end()); + auto decoded = readManifestSection(duplicate); + ASSERT_FALSE(decoded.has_value()); + EXPECT_NE(decoded.error().find("multiple parser-module manifest sections"), std::string::npos); + EXPECT_FALSE(appendManifestSection(kEmbeddedGolden, kManifest).has_value()); +} + +TEST(ParserModuleManifest, RejectsTruncatedPreamblesAndSections) { + for (size_t size = 0; size < kMinimalWasm.size(); ++size) { + EXPECT_FALSE(readManifestSection(Span(kMinimalWasm.data(), size)).has_value()) << size; + } + + auto wrong_magic = kMinimalWasm; + wrong_magic[1] = 0; + EXPECT_FALSE(readManifestSection(wrong_magic).has_value()); + + for (size_t size = kMinimalWasm.size(); size < kEmbeddedGolden.size(); ++size) { + EXPECT_FALSE(readManifestSection(Span(kEmbeddedGolden.data(), size)).has_value()) << size; + } + + std::vector overflowing_leb(kMinimalWasm.begin(), kMinimalWasm.end()); + overflowing_leb.insert(overflowing_leb.end(), {0, 0x80, 0x80, 0x80, 0x80, 0x10}); + EXPECT_FALSE(readManifestSection(overflowing_leb).has_value()); +} + +} // namespace +} // namespace PJ::parser_module diff --git a/pj_base/tests/parser_module_object_writer_test.cpp b/pj_base/tests/parser_module_object_writer_test.cpp new file mode 100644 index 00000000..428f5ebe --- /dev/null +++ b/pj_base/tests/parser_module_object_writer_test.cpp @@ -0,0 +1,580 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin/builtin_object_codec.hpp" +#include "pj_base/builtin/compressed_point_cloud.hpp" +#include "pj_base/builtin/depth_image.hpp" +#include "pj_base/builtin/image.hpp" +#include "pj_base/builtin/mesh3d.hpp" +#include "pj_base/builtin/occupancy_grid.hpp" +#include "pj_base/builtin/occupancy_grid_update.hpp" +#include "pj_base/builtin/point_cloud.hpp" +#include "pj_base/builtin/video_frame.hpp" +#include "pj_base/builtin/voxel_grid.hpp" +#include "pj_base/parser_module/module.hpp" +#include "pj_base/parser_module_abi.h" + +namespace { + +class TokenAndTimestampParser final : public pj::FunctionalParser { + public: + pj::Status bind(const pj::BindingInfo&) override { + return pj::Status::ok(); + } + + pj::Status parseScalars(pj::PayloadView, pj::Timestamp timestamp, pj::ScalarWriter& output) override { + saw_timestamp = timestamp.has_value; + timestamp_ns = timestamp.nanoseconds; + return output.add("value", uint64_t{1}); + } + + static inline bool saw_timestamp = false; + static inline int64_t timestamp_ns = 0; +}; + +class FinishFailureParser final : public pj::FunctionalParser { + public: + pj::Status bind(const pj::BindingInfo&) override { + return pj::Status::ok(); + } + + pj::Status parseObject(pj::PayloadView, pj::Timestamp, pj::ObjectWriter&) override { + return pj::Status::ok(); + } +}; + +PJ::Span hostBytes(std::string_view value) { + return {reinterpret_cast(value.data()), value.size()}; +} + +std::string_view kitText(pj::ByteView value) { + return {reinterpret_cast(value.data), value.size}; +} + +TEST(ParserModuleAuthoringCodec, KitDecodesHostBindingAndParseFramingByteForByte) { + const std::array schema{0x00, 0x7F, 0xFF}; + const PJ::parser_module::BindingInfoV1 binding{ + .route = PJ::parser_module::Route::kObject, + .claim_index = UINT32_C(0x01020304), + .expected_object_type = static_cast(PJ::sdk::BuiltinObjectType::kPointCloud), + .encoding = hostBytes("cdr"), + .type_name = hostBytes("example/Type"), + .schema = schema, + .claim_id = hostBytes("claim"), + .config_json = hostBytes("{}"), + .schema_digest = hostBytes("digest"), + }; + auto encoded_binding = PJ::parser_module::writeBindingInfoV1(binding); + ASSERT_TRUE(encoded_binding.has_value()) << encoded_binding.error(); + auto decoded_binding = pj::readBindingInfo({encoded_binding->data(), encoded_binding->size()}); + ASSERT_TRUE(decoded_binding.hasValue()) << decoded_binding.status().message(); + EXPECT_EQ(decoded_binding->route(), pj::Route::kObject); + EXPECT_EQ(decoded_binding->claimIndex(), UINT32_C(0x01020304)); + EXPECT_EQ(decoded_binding->expectedObjectType(), static_cast(PJ::sdk::BuiltinObjectType::kPointCloud)); + EXPECT_EQ(kitText(decoded_binding->encoding()), "cdr"); + EXPECT_EQ(kitText(decoded_binding->typeName()), "example/Type"); + EXPECT_EQ(decoded_binding->schema().size, schema.size()); + EXPECT_EQ(kitText(decoded_binding->claimId()), "claim"); + EXPECT_EQ(kitText(decoded_binding->configJson()), "{}"); + EXPECT_EQ(kitText(decoded_binding->schemaDigest()), "digest"); + auto owned_binding = decoded_binding->owningCopy(); + ASSERT_TRUE(owned_binding.hasValue()) << owned_binding.status().message(); + encoded_binding->assign(encoded_binding->size(), 0); + EXPECT_EQ(kitText(owned_binding->encoding()), "cdr"); + EXPECT_EQ(kitText(owned_binding->schema()), std::string_view("\0\x7f\xff", 3)); + + const std::array payload{0xAA, 0x00, 0xFF}; + auto encoded_input = + PJ::parser_module::writeParseInputV1({.has_timestamp = true, .timestamp_ns = -2, .payload = payload}); + ASSERT_TRUE(encoded_input.has_value()) << encoded_input.error(); + auto decoded_input = pj::readParseInput({encoded_input->data(), encoded_input->size()}); + ASSERT_TRUE(decoded_input.hasValue()) << decoded_input.status().message(); + EXPECT_TRUE(decoded_input->has_timestamp); + EXPECT_EQ(decoded_input->timestamp_ns, -2); + ASSERT_EQ(decoded_input->payload.size, payload.size()); + EXPECT_EQ( + std::vector(decoded_input->payload.data, decoded_input->payload.data + decoded_input->payload.size), + std::vector(payload.begin(), payload.end())); +} + +TEST(ParserModuleExports, ThreadsPerMessageTimestampAndRejectsStaleGenerationalTokens) { + using Exports = pj::detail::ModuleExports; + const uint64_t first = Exports::create(0); + ASSERT_NE(first, 0U); + Exports::destroy(first); + const uint64_t second = Exports::create(0); + ASSERT_NE(second, 0U); + EXPECT_NE(first, second); + EXPECT_EQ(Exports::bind(first, 0, 0), pj::kModuleBadToken); + std::array error{}; + const uint64_t written = + Exports::lastError(first, pj::detail::addressOf(error.data()), static_cast(error.size())); + EXPECT_NE( + std::string_view(error.data(), static_cast(written)).find("stale or unknown"), std::string_view::npos); + + const PJ::parser_module::BindingInfoV1 binding{ + .route = PJ::parser_module::Route::kScalar, + .claim_index = 0, + .expected_object_type = 0, + .encoding = {}, + .type_name = {}, + .schema = {}, + .claim_id = {}, + .config_json = {}, + .schema_digest = {}, + }; + auto binding_bytes = PJ::parser_module::writeBindingInfoV1(binding); + ASSERT_TRUE(binding_bytes.has_value()) << binding_bytes.error(); + ASSERT_EQ(Exports::bind(second, pj::detail::addressOf(binding_bytes->data()), binding_bytes->size()), pj::kModuleOk); + const std::array payload{5}; + auto parse_bytes = + PJ::parser_module::writeParseInputV1({.has_timestamp = true, .timestamp_ns = -123, .payload = payload}); + ASSERT_TRUE(parse_bytes.has_value()) << parse_bytes.error(); + uint64_t output_address = 0; + uint64_t output_length = 0; + ASSERT_EQ( + Exports::parse( + second, pj::detail::addressOf(parse_bytes->data()), parse_bytes->size(), + pj::detail::addressOf(&output_address), pj::detail::addressOf(&output_length)), + pj::kModuleOk); + EXPECT_TRUE(TokenAndTimestampParser::saw_timestamp); + EXPECT_EQ(TokenAndTimestampParser::timestamp_ns, -123); + EXPECT_NE(output_address, 0U); + EXPECT_NE(output_length, 0U); + Exports::destroy(second); +} + +TEST(ParserModuleExports, InstanceTableAllowsParsingWhileOtherTokensChange) { + using Exports = pj::detail::ModuleExports; + const uint64_t parsing_token = Exports::create(0); + ASSERT_NE(parsing_token, 0U); + const PJ::parser_module::BindingInfoV1 binding{ + .route = PJ::parser_module::Route::kScalar, + .claim_index = 0, + .expected_object_type = 0, + .encoding = {}, + .type_name = {}, + .schema = {}, + .claim_id = {}, + .config_json = {}, + .schema_digest = {}, + }; + auto binding_bytes = PJ::parser_module::writeBindingInfoV1(binding); + ASSERT_TRUE(binding_bytes.has_value()); + ASSERT_EQ( + Exports::bind(parsing_token, pj::detail::addressOf(binding_bytes->data()), binding_bytes->size()), pj::kModuleOk); + const std::array payload{1}; + auto parse_bytes = PJ::parser_module::writeParseInputV1({.payload = payload}); + ASSERT_TRUE(parse_bytes.has_value()); + + int32_t parse_result = pj::kModuleError; + std::thread worker([&] { + for (size_t index = 0; index < 200; ++index) { + uint64_t output_address = 0; + uint64_t output_length = 0; + parse_result = Exports::parse( + parsing_token, pj::detail::addressOf(parse_bytes->data()), parse_bytes->size(), + pj::detail::addressOf(&output_address), pj::detail::addressOf(&output_length)); + if (parse_result != pj::kModuleOk) { + return; + } + } + }); + for (size_t index = 0; index < 200; ++index) { + const uint64_t other = Exports::create(0); + EXPECT_NE(other, 0U); + if (other != 0) { + Exports::destroy(other); + } + } + worker.join(); + EXPECT_EQ(parse_result, pj::kModuleOk); + Exports::destroy(parsing_token); +} + +TEST(ParserModuleExports, WriterFinishFailureUsesGenericErrorCode) { + using Exports = pj::detail::ModuleExports; + const uint64_t token = Exports::create(0); + ASSERT_NE(token, 0U); + const PJ::parser_module::BindingInfoV1 binding{ + .route = PJ::parser_module::Route::kObject, + .claim_index = 0, + .expected_object_type = 3, + .encoding = {}, + .type_name = {}, + .schema = {}, + .claim_id = {}, + .config_json = {}, + .schema_digest = {}, + }; + auto binding_bytes = PJ::parser_module::writeBindingInfoV1(binding); + ASSERT_TRUE(binding_bytes.has_value()); + ASSERT_EQ(Exports::bind(token, pj::detail::addressOf(binding_bytes->data()), binding_bytes->size()), pj::kModuleOk); + const std::array payload{1}; + auto parse_bytes = PJ::parser_module::writeParseInputV1({.payload = payload}); + ASSERT_TRUE(parse_bytes.has_value()); + uint64_t output_address = 0; + uint64_t output_length = 0; + EXPECT_EQ( + Exports::parse( + token, pj::detail::addressOf(parse_bytes->data()), parse_bytes->size(), + pj::detail::addressOf(&output_address), pj::detail::addressOf(&output_length)), + pj::kModuleError); + Exports::destroy(token); +} + +TEST(ParserModuleObjectWriter, PointCloudFullWireMatchesGoldenDescriptorAndHostCodec) { + pj::ObjectWriter writer; + auto cloud = writer.pointCloud(); + ASSERT_TRUE(cloud.setTimestamp(0).isOk()); + ASSERT_TRUE(cloud.setWidth(1).isOk()); + ASSERT_TRUE(cloud.setHeight(1).isOk()); + ASSERT_TRUE(cloud.setPointStep(1).isOk()); + ASSERT_TRUE(cloud.setRowStep(2).isOk()); + ASSERT_TRUE(cloud.setBigEndian(false).isOk()); + ASSERT_TRUE(cloud.setDense(true).isOk()); + const std::array data{0xAA, 0xBB}; + ASSERT_TRUE(cloud.setData({data.data(), data.size()}).isOk()); + ASSERT_TRUE(cloud.setFrameId("f").isOk()); + auto descriptor = writer.finish(); + ASSERT_TRUE(descriptor.hasValue()) << descriptor.status().message(); + + const std::vector expected{ + 0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x0A, 0x04, 0x08, 0x00, 0x10, 0x00, 0x10, 0x01, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x02, 0x30, 0x00, 0x38, 0x01, 0x4A, 0x02, 0xAA, 0xBB, 0x52, 0x01, 0x66, + }; + EXPECT_EQ(std::vector(descriptor->data(), descriptor->data() + descriptor->size()), expected); + + auto decoded = PJ::parser_module::readOutputDescriptorV1({descriptor->data(), descriptor->size()}); + ASSERT_TRUE(decoded.has_value()) << decoded.error(); + const auto* object = std::get_if(&*decoded); + ASSERT_NE(object, nullptr); + auto canonical = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kPointCloud, object->wire.data(), object->wire.size()); + ASSERT_TRUE(canonical.has_value()) << canonical.error(); + const auto* result = std::any_cast(&*canonical); + ASSERT_NE(result, nullptr); + EXPECT_EQ(result->data.size(), 2U); + EXPECT_EQ(result->data[1], 0xBB); + EXPECT_EQ(result->frame_id, "f"); +} + +TEST(ParserModuleObjectWriter, PointCloudAndImageSplicesUseFrozenEligibleFields) { + const std::array payload{}; + pj::ObjectWriter cloud_writer({payload.data(), payload.size()}); + auto cloud = cloud_writer.pointCloud(); + ASSERT_TRUE(cloud.setWidth(2).isOk()); + ASSERT_TRUE(cloud.setDataFromInput({11, 8}).isOk()); + auto cloud_descriptor = cloud_writer.finish(); + ASSERT_TRUE(cloud_descriptor.hasValue()); + auto decoded_cloud = PJ::parser_module::readOutputDescriptorV1({cloud_descriptor->data(), cloud_descriptor->size()}); + ASSERT_TRUE(decoded_cloud.has_value()) << decoded_cloud.error(); + const auto& cloud_output = std::get(*decoded_cloud); + ASSERT_TRUE(cloud_output.splice.has_value()); + EXPECT_EQ(cloud_output.splice->field_number, 9U); + EXPECT_EQ(cloud_output.splice->input_offset, 11U); + EXPECT_EQ(cloud_output.splice->input_length, 8U); + + pj::ObjectWriter image_writer({payload.data(), payload.size()}); + auto image = image_writer.image(); + ASSERT_TRUE(image.setWidth(4).isOk()); + ASSERT_TRUE(image.setHeight(3).isOk()); + ASSERT_TRUE(image.setEncoding("mono8").isOk()); + ASSERT_TRUE(image.setDataFromInput({5, 12}).isOk()); + auto image_descriptor = image_writer.finish(); + ASSERT_TRUE(image_descriptor.hasValue()); + auto decoded_image = PJ::parser_module::readOutputDescriptorV1({image_descriptor->data(), image_descriptor->size()}); + ASSERT_TRUE(decoded_image.has_value()) << decoded_image.error(); + const auto& image_output = std::get(*decoded_image); + ASSERT_TRUE(image_output.splice.has_value()); + EXPECT_EQ(image_output.object_type, 1U); + EXPECT_EQ(image_output.splice->field_number, 7U); +} + +TEST(ParserModuleObjectWriter, ImageFullWireDecodesWithHostCodec) { + pj::ObjectWriter writer; + auto image = writer.image(); + ASSERT_TRUE(image.setTimestamp(-1).isOk()); + ASSERT_TRUE(image.setWidth(2).isOk()); + ASSERT_TRUE(image.setHeight(1).isOk()); + ASSERT_TRUE(image.setEncoding("mono8").isOk()); + ASSERT_TRUE(image.setRowStep(2).isOk()); + ASSERT_TRUE(image.setBigEndian(false).isOk()); + ASSERT_TRUE(image.setFrameId("camera").isOk()); + ASSERT_TRUE(image.setCompressedDepthMin(0.5F).isOk()); + ASSERT_TRUE(image.setCompressedDepthMax(3.5F).isOk()); + const std::array data{0x10, 0x20}; + ASSERT_TRUE(image.setData({data.data(), data.size()}).isOk()); + + auto descriptor = writer.finish(); + ASSERT_TRUE(descriptor.hasValue()) << descriptor.status().message(); + auto decoded = PJ::parser_module::readOutputDescriptorV1({descriptor->data(), descriptor->size()}); + ASSERT_TRUE(decoded.has_value()) << decoded.error(); + const auto& object = std::get(*decoded); + EXPECT_FALSE(object.splice.has_value()); + auto canonical = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kImage, object.wire.data(), object.wire.size()); + ASSERT_TRUE(canonical.has_value()) << canonical.error(); + const auto* result = std::any_cast(&*canonical); + ASSERT_NE(result, nullptr); + EXPECT_EQ(result->timestamp_ns, -1); + EXPECT_EQ(result->width, 2U); + EXPECT_EQ(result->height, 1U); + EXPECT_EQ(result->encoding, "mono8"); + EXPECT_EQ(result->data.size(), 2U); + EXPECT_EQ(result->data[1], 0x20); + EXPECT_EQ(result->compressed_depth_min, 0.5F); + EXPECT_EQ(result->compressed_depth_max, 3.5F); + EXPECT_EQ(result->frame_id, "camera"); +} + +TEST(ParserModuleObjectWriter, ScalarWriterUsesHostReadableDescriptor) { + pj::ScalarWriter writer; + ASSERT_TRUE(writer.setTimestamp(44).isOk()); + ASSERT_TRUE(writer.add("temperature", 21.5).isOk()); + ASSERT_TRUE(writer.add("ready", true).isOk()); + ASSERT_TRUE(writer.add("label", "ok").isOk()); + auto descriptor = writer.finish(); + ASSERT_TRUE(descriptor.hasValue()) << descriptor.status().message(); + auto decoded = PJ::parser_module::readOutputDescriptorV1({descriptor->data(), descriptor->size()}); + ASSERT_TRUE(decoded.has_value()) << decoded.error(); + const auto& scalar = std::get(*decoded); + ASSERT_EQ(scalar.fields.size(), 3U); + EXPECT_EQ(scalar.timestamp_ns, 44); + EXPECT_EQ(scalar.fields[2].name, "label"); + EXPECT_EQ(std::get(scalar.fields[2].value), "ok"); +} + +TEST(ParserModuleObjectWriter, AllAdditionalSpliceEligibleBuildersRoundTripFullWire) { + const std::array data{7, 8, 9}; + + pj::ObjectWriter depth_writer; + auto depth = depth_writer.depthImage(); + ASSERT_TRUE(depth.setTimestamp(11).isOk()); + ASSERT_TRUE(depth.setWidth(2).isOk()); + ASSERT_TRUE(depth.setHeight(3).isOk()); + ASSERT_TRUE(depth.setEncoding("16UC1").isOk()); + ASSERT_TRUE(depth.setIntrinsics({1, 0, 2, 0, 3, 4, 0, 0, 1}).isOk()); + ASSERT_TRUE(depth.setDistortionModel("plumb_bob").isOk()); + ASSERT_TRUE(depth.addDistortionCoefficient(0.25).isOk()); + ASSERT_TRUE(depth.setData({data.data(), data.size()}).isOk()); + auto depth_descriptor = depth_writer.finish(); + ASSERT_TRUE(depth_descriptor.hasValue()) << depth_descriptor.status().message(); + auto depth_output = PJ::parser_module::readOutputDescriptorV1({depth_descriptor->data(), depth_descriptor->size()}); + ASSERT_TRUE(depth_output.has_value()) << depth_output.error(); + const auto& depth_wire = std::get(*depth_output).wire; + auto decoded_depth = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kDepthImage, depth_wire.data(), depth_wire.size()); + ASSERT_TRUE(decoded_depth.has_value()) << decoded_depth.error(); + const auto* depth_value = std::any_cast(&*decoded_depth); + ASSERT_NE(depth_value, nullptr); + EXPECT_EQ(depth_value->data[2], 9U); + EXPECT_EQ(depth_value->K[4], 3.0); + ASSERT_EQ(depth_value->D.size(), 1U); + + pj::ObjectWriter grid_writer; + auto grid = grid_writer.occupancyGrid(); + ASSERT_TRUE(grid.setTimestamp(12).isOk()); + ASSERT_TRUE(grid.setFrameId("map").isOk()); + ASSERT_TRUE(grid.setOrigin(1, 2, 3).isOk()); + ASSERT_TRUE(grid.setResolution(0.5).isOk()); + ASSERT_TRUE(grid.setWidth(3).isOk()); + ASSERT_TRUE(grid.setHeight(1).isOk()); + ASSERT_TRUE(grid.setData({data.data(), data.size()}).isOk()); + auto grid_descriptor = grid_writer.finish(); + ASSERT_TRUE(grid_descriptor.hasValue()) << grid_descriptor.status().message(); + auto grid_output = PJ::parser_module::readOutputDescriptorV1({grid_descriptor->data(), grid_descriptor->size()}); + ASSERT_TRUE(grid_output.has_value()) << grid_output.error(); + const auto& grid_wire = std::get(*grid_output).wire; + auto decoded_grid = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kOccupancyGrid, grid_wire.data(), grid_wire.size()); + ASSERT_TRUE(decoded_grid.has_value()) << decoded_grid.error(); + const auto* grid_value = std::any_cast(&*decoded_grid); + ASSERT_NE(grid_value, nullptr); + EXPECT_EQ(grid_value->frame_id, "map"); + EXPECT_EQ(grid_value->data.size(), 3U); + + pj::ObjectWriter compressed_writer; + auto compressed = compressed_writer.compressedPointCloud(); + ASSERT_TRUE(compressed.setTimestamp(13).isOk()); + ASSERT_TRUE(compressed.setFrameId("lidar").isOk()); + ASSERT_TRUE(compressed.setFormat("draco").isOk()); + ASSERT_TRUE(compressed.setData({data.data(), data.size()}).isOk()); + auto compressed_descriptor = compressed_writer.finish(); + ASSERT_TRUE(compressed_descriptor.hasValue()) << compressed_descriptor.status().message(); + auto compressed_output = + PJ::parser_module::readOutputDescriptorV1({compressed_descriptor->data(), compressed_descriptor->size()}); + ASSERT_TRUE(compressed_output.has_value()) << compressed_output.error(); + const auto& compressed_wire = std::get(*compressed_output).wire; + auto decoded_compressed = PJ::deserializeBuiltinObject( + PJ::sdk::BuiltinObjectType::kCompressedPointCloud, compressed_wire.data(), compressed_wire.size()); + ASSERT_TRUE(decoded_compressed.has_value()) << decoded_compressed.error(); + const auto* compressed_value = std::any_cast(&*decoded_compressed); + ASSERT_NE(compressed_value, nullptr); + EXPECT_EQ(compressed_value->format, "draco"); + EXPECT_EQ(compressed_value->data[0], 7U); + + pj::ObjectWriter mesh_writer; + auto mesh = mesh_writer.mesh3D(); + ASSERT_TRUE(mesh.setTimestamp(14).isOk()); + ASSERT_TRUE(mesh.setFrameId("world").isOk()); + ASSERT_TRUE(mesh.setId("mesh").isOk()); + ASSERT_TRUE(mesh.setPose(1, 2, 3).isOk()); + ASSERT_TRUE(mesh.setScale(2, 3, 4).isOk()); + ASSERT_TRUE(mesh.setFormat("glb").isOk()); + ASSERT_TRUE(mesh.setData({data.data(), data.size()}).isOk()); + ASSERT_TRUE(mesh.setColor(1, 0.5, 0, 1).isOk()); + ASSERT_TRUE(mesh.setOverrideColor(true).isOk()); + auto mesh_descriptor = mesh_writer.finish(); + ASSERT_TRUE(mesh_descriptor.hasValue()) << mesh_descriptor.status().message(); + auto mesh_output = PJ::parser_module::readOutputDescriptorV1({mesh_descriptor->data(), mesh_descriptor->size()}); + ASSERT_TRUE(mesh_output.has_value()) << mesh_output.error(); + const auto& mesh_wire = std::get(*mesh_output).wire; + auto decoded_mesh = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kMesh3D, mesh_wire.data(), mesh_wire.size()); + ASSERT_TRUE(decoded_mesh.has_value()) << decoded_mesh.error(); + const auto* mesh_value = std::any_cast(&*decoded_mesh); + ASSERT_NE(mesh_value, nullptr); + EXPECT_EQ(mesh_value->data.size(), 3U); + EXPECT_EQ(mesh_value->scale.y, 3.0); + EXPECT_TRUE(mesh_value->override_color); + + pj::ObjectWriter video_writer; + auto video = video_writer.videoFrame(); + ASSERT_TRUE(video.setTimestamp(15).isOk()); + ASSERT_TRUE(video.setFrameId("camera").isOk()); + ASSERT_TRUE(video.setFormat("h264").isOk()); + ASSERT_TRUE(video.setData({data.data(), data.size()}).isOk()); + auto video_descriptor = video_writer.finish(); + ASSERT_TRUE(video_descriptor.hasValue()) << video_descriptor.status().message(); + auto video_output = PJ::parser_module::readOutputDescriptorV1({video_descriptor->data(), video_descriptor->size()}); + ASSERT_TRUE(video_output.has_value()) << video_output.error(); + const auto& video_wire = std::get(*video_output).wire; + auto decoded_video = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kVideoFrame, video_wire.data(), video_wire.size()); + ASSERT_TRUE(decoded_video.has_value()) << decoded_video.error(); + const auto* video_value = std::any_cast(&*decoded_video); + ASSERT_NE(video_value, nullptr); + EXPECT_EQ(video_value->format, "h264"); + EXPECT_EQ(video_value->data[1], 8U); + + pj::ObjectWriter update_writer; + auto update = update_writer.occupancyGridUpdate(); + ASSERT_TRUE(update.setTimestamp(16).isOk()); + ASSERT_TRUE(update.setFrameId("map").isOk()); + ASSERT_TRUE(update.setX(-2).isOk()); + ASSERT_TRUE(update.setY(4).isOk()); + ASSERT_TRUE(update.setWidth(3).isOk()); + ASSERT_TRUE(update.setHeight(1).isOk()); + ASSERT_TRUE(update.setData({data.data(), data.size()}).isOk()); + auto update_descriptor = update_writer.finish(); + ASSERT_TRUE(update_descriptor.hasValue()) << update_descriptor.status().message(); + auto update_output = + PJ::parser_module::readOutputDescriptorV1({update_descriptor->data(), update_descriptor->size()}); + ASSERT_TRUE(update_output.has_value()) << update_output.error(); + const auto& update_wire = std::get(*update_output).wire; + auto decoded_update = PJ::deserializeBuiltinObject( + PJ::sdk::BuiltinObjectType::kOccupancyGridUpdate, update_wire.data(), update_wire.size()); + ASSERT_TRUE(decoded_update.has_value()) << decoded_update.error(); + const auto* update_value = std::any_cast(&*decoded_update); + ASSERT_NE(update_value, nullptr); + EXPECT_EQ(update_value->x, -2); + EXPECT_EQ(update_value->data[2], 9U); + + pj::ObjectWriter voxel_writer; + auto voxel = voxel_writer.voxelGrid(); + ASSERT_TRUE(voxel.setTimestamp(17).isOk()); + ASSERT_TRUE(voxel.setFrameId("map").isOk()); + ASSERT_TRUE(voxel.setOrigin(1, 2, 3).isOk()); + ASSERT_TRUE(voxel.setCellSize(0.1, 0.2, 0.3).isOk()); + ASSERT_TRUE(voxel.setColumnCount(3).isOk()); + ASSERT_TRUE(voxel.setRowCount(1).isOk()); + ASSERT_TRUE(voxel.setSliceCount(1).isOk()); + ASSERT_TRUE(voxel.setCellStride(1).isOk()); + ASSERT_TRUE(voxel.setRowStride(3).isOk()); + ASSERT_TRUE(voxel.setSliceStride(3).isOk()); + ASSERT_TRUE(voxel.addField("occupancy", 0, pj::ObjectWriter::PointFieldDatatype::kUint8).isOk()); + ASSERT_TRUE(voxel.setData({data.data(), data.size()}).isOk()); + auto voxel_descriptor = voxel_writer.finish(); + ASSERT_TRUE(voxel_descriptor.hasValue()) << voxel_descriptor.status().message(); + auto voxel_output = PJ::parser_module::readOutputDescriptorV1({voxel_descriptor->data(), voxel_descriptor->size()}); + ASSERT_TRUE(voxel_output.has_value()) << voxel_output.error(); + const auto& voxel_wire = std::get(*voxel_output).wire; + auto decoded_voxel = + PJ::deserializeBuiltinObject(PJ::sdk::BuiltinObjectType::kVoxelGrid, voxel_wire.data(), voxel_wire.size()); + ASSERT_TRUE(decoded_voxel.has_value()) << decoded_voxel.error(); + const auto* voxel_value = std::any_cast(&*decoded_voxel); + ASSERT_NE(voxel_value, nullptr); + ASSERT_EQ(voxel_value->fields.size(), 1U); + EXPECT_EQ(voxel_value->fields[0].name, "occupancy"); + EXPECT_EQ(voxel_value->data.size(), 3U); +} + +TEST(ParserModuleObjectWriter, EveryEligibleBuilderSupportsValidatedSpliceOutput) { + const std::array payload{}; + const auto expect_splice = [](pj::ObjectWriter& writer, uint16_t type, uint32_t field) { + auto descriptor = writer.finish(); + ASSERT_TRUE(descriptor.hasValue()) << descriptor.status().message(); + auto decoded = PJ::parser_module::readOutputDescriptorV1({descriptor->data(), descriptor->size()}); + ASSERT_TRUE(decoded.has_value()) << decoded.error(); + const auto& object = std::get(*decoded); + EXPECT_EQ(object.object_type, type); + ASSERT_TRUE(object.splice.has_value()); + EXPECT_EQ(object.splice->field_number, field); + EXPECT_EQ(object.splice->input_offset, 2U); + EXPECT_EQ(object.splice->input_length, 3U); + }; + + pj::ObjectWriter depth({payload.data(), payload.size()}); + ASSERT_TRUE(depth.depthImage().setDataFromInput({2, 3}).isOk()); + expect_splice(depth, 4, 5); + pj::ObjectWriter grid({payload.data(), payload.size()}); + ASSERT_TRUE(grid.occupancyGrid().setDataFromInput({2, 3}).isOk()); + expect_splice(grid, 7, 7); + pj::ObjectWriter compressed({payload.data(), payload.size()}); + ASSERT_TRUE(compressed.compressedPointCloud().setDataFromInput({2, 3}).isOk()); + expect_splice(compressed, 8, 4); + pj::ObjectWriter mesh({payload.data(), payload.size()}); + ASSERT_TRUE(mesh.mesh3D().setDataFromInput({2, 3}).isOk()); + expect_splice(mesh, 9, 7); + pj::ObjectWriter video({payload.data(), payload.size()}); + ASSERT_TRUE(video.videoFrame().setDataFromInput({2, 3}).isOk()); + expect_splice(video, 10, 3); + pj::ObjectWriter update({payload.data(), payload.size()}); + ASSERT_TRUE(update.occupancyGridUpdate().setDataFromInput({2, 3}).isOk()); + expect_splice(update, 15, 7); + pj::ObjectWriter voxel({payload.data(), payload.size()}); + ASSERT_TRUE(voxel.voxelGrid().setDataFromInput({2, 3}).isOk()); + expect_splice(voxel, 18, 12); +} + +TEST(ParserModuleObjectWriter, RejectsConflictingOrOutOfBoundsBulkDataSelection) { + const std::array payload{1, 2}; + pj::ObjectWriter copied({payload.data(), payload.size()}); + auto image = copied.image(); + ASSERT_TRUE(image.setData({payload.data(), payload.size()}).isOk()); + EXPECT_FALSE(image.setDataFromInput({0, 1}).isOk()); + EXPECT_NE(copied.status().message().find("contract_violation"), std::string_view::npos); + + pj::ObjectWriter spliced({payload.data(), payload.size()}); + auto cloud = spliced.pointCloud(); + ASSERT_TRUE(cloud.setDataFromInput({0, 1}).isOk()); + EXPECT_FALSE(cloud.setDataFromInput({1, 1}).isOk()); + + pj::ObjectWriter out_of_bounds({payload.data(), payload.size()}); + EXPECT_FALSE(out_of_bounds.videoFrame().setDataFromInput({1, 2}).isOk()); +} + +} // namespace diff --git a/pj_base/tests/parser_module_wasm_audit.cpp b/pj_base/tests/parser_module_wasm_audit.cpp new file mode 100644 index 00000000..d299db50 --- /dev/null +++ b/pj_base/tests/parser_module_wasm_audit.cpp @@ -0,0 +1,554 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module_manifest.hpp" + +namespace { + +using PJ::Expected; +using PJ::Span; +using PJ::unexpected; + +constexpr std::array kWasmPreamble{0x00, 0x61, 0x73, 0x6D, 0x01, 0x00, 0x00, 0x00}; +constexpr uint8_t kI32 = 0x7F; +constexpr uint8_t kI64 = 0x7E; + +class Cursor { + public: + explicit Cursor(Span bytes) : bytes_(bytes) {} + + [[nodiscard]] bool empty() const noexcept { + return position_ == bytes_.size(); + } + + [[nodiscard]] size_t remaining() const noexcept { + return position_ <= bytes_.size() ? bytes_.size() - position_ : 0; + } + + [[nodiscard]] Expected byte() { + if (position_ >= bytes_.size()) { + return unexpected(std::string("truncated wasm byte")); + } + return bytes_[position_++]; + } + + [[nodiscard]] Expected varUint32() { + uint32_t value = 0; + for (size_t index = 0; index < 5; ++index) { + auto next = byte(); + if (!next) { + return unexpected(next.error()); + } + if (index == 4 && (*next & UINT8_C(0xF0)) != 0) { + return unexpected(std::string("wasm varuint32 overflows uint32")); + } + value |= static_cast(*next & UINT8_C(0x7F)) << (index * 7U); + if ((*next & UINT8_C(0x80)) == 0) { + return value; + } + } + return unexpected(std::string("wasm varuint32 exceeds five bytes")); + } + + [[nodiscard]] Expected name() { + auto length = varUint32(); + if (!length) { + return unexpected(length.error()); + } + if (static_cast(*length) > remaining()) { + return unexpected(std::string("wasm name exceeds the remaining section bytes")); + } + const char* begin = reinterpret_cast(bytes_.data() + position_); + position_ += *length; + return std::string(begin, *length); + } + + [[nodiscard]] Expected take(uint32_t size) { + if (static_cast(size) > remaining()) { + return unexpected(std::string("wasm section exceeds the remaining module bytes")); + } + Cursor result(bytes_.subspan(position_, size)); + position_ += size; + return result; + } + + private: + Span bytes_; + size_t position_ = 0; +}; + +struct FunctionType { + std::vector parameters; + std::vector results; +}; + +struct Export { + std::string name; + uint8_t kind = 0; + uint32_t index = 0; +}; + +struct ModuleInfo { + std::vector types; + std::vector function_types; + std::vector exports; + bool has_start_section = false; + size_t section_count = 0; +}; + +[[nodiscard]] bool validValueType(uint8_t value) { + switch (value) { + case 0x7F: // i32 + case 0x7E: // i64 + case 0x7D: // f32 + case 0x7C: // f64 + case 0x7B: // v128 + case 0x70: // funcref + case 0x6F: // externref + return true; + default: + return false; + } +} + +[[nodiscard]] Expected> readValueTypes(Cursor* cursor) { + auto count = cursor->varUint32(); + if (!count) { + return unexpected(count.error()); + } + try { + std::vector result; + result.reserve(*count); + for (uint32_t index = 0; index < *count; ++index) { + auto value = cursor->byte(); + if (!value) { + return unexpected(value.error()); + } + if (!validValueType(*value)) { + return unexpected(std::string("wasm function type contains an invalid value type")); + } + result.push_back(*value); + } + return result; + } catch (const std::bad_alloc&) { + return unexpected(std::string("allocation failed while reading wasm value types")); + } +} + +[[nodiscard]] Expected requireConsumed(const Cursor& cursor, std::string_view section) { + if (!cursor.empty()) { + return unexpected(std::string(section) + " section contains trailing bytes"); + } + return {}; +} + +[[nodiscard]] Expected parseTypeSection(Cursor cursor, ModuleInfo* module) { + auto count = cursor.varUint32(); + if (!count) { + return unexpected(count.error()); + } + try { + module->types.reserve(*count); + for (uint32_t index = 0; index < *count; ++index) { + auto form = cursor.byte(); + if (!form || *form != UINT8_C(0x60)) { + return unexpected(std::string("wasm type section contains a non-function type")); + } + auto parameters = readValueTypes(&cursor); + auto results = readValueTypes(&cursor); + if (!parameters) { + return unexpected(parameters.error()); + } + if (!results) { + return unexpected(results.error()); + } + module->types.push_back(FunctionType{std::move(*parameters), std::move(*results)}); + } + } catch (const std::bad_alloc&) { + return unexpected(std::string("allocation failed while reading the wasm type section")); + } + return requireConsumed(cursor, "type"); +} + +[[nodiscard]] Expected readLimits(Cursor* cursor) { + auto flags = cursor->varUint32(); + auto minimum = cursor->varUint32(); + if (!flags || !minimum) { + return unexpected(std::string("truncated wasm limits")); + } + if (*flags > 1) { + return unexpected(std::string("unsupported wasm limits flags")); + } + if ((*flags & 1U) != 0) { + auto maximum = cursor->varUint32(); + if (!maximum) { + return unexpected(maximum.error()); + } + } + return {}; +} + +[[nodiscard]] Expected parseImportSection(Cursor cursor, ModuleInfo* module) { + auto count = cursor.varUint32(); + if (!count) { + return unexpected(count.error()); + } + for (uint32_t index = 0; index < *count; ++index) { + auto module_name = cursor.name(); + auto field_name = cursor.name(); + auto kind = cursor.byte(); + if (!module_name || !field_name || !kind) { + return unexpected(std::string("truncated wasm import entry")); + } + switch (*kind) { + case 0: { + auto type_index = cursor.varUint32(); + if (!type_index) { + return unexpected(type_index.error()); + } + module->function_types.push_back(*type_index); + break; + } + case 1: { + auto element_type = cursor.byte(); + if (!element_type || (*element_type != UINT8_C(0x70) && *element_type != UINT8_C(0x6F))) { + return unexpected(std::string("invalid wasm table import")); + } + auto limits = readLimits(&cursor); + if (!limits) { + return unexpected(limits.error()); + } + break; + } + case 2: { + auto limits = readLimits(&cursor); + if (!limits) { + return unexpected(limits.error()); + } + break; + } + case 3: { + auto value_type = cursor.byte(); + auto mutability = cursor.byte(); + if (!value_type || !mutability || !validValueType(*value_type) || *mutability > 1) { + return unexpected(std::string("invalid wasm global import")); + } + break; + } + case 4: { + auto attribute = cursor.varUint32(); + auto type_index = cursor.varUint32(); + if (!attribute || !type_index) { + return unexpected(std::string("truncated wasm tag import")); + } + break; + } + default: + return unexpected(std::string("unknown wasm import kind")); + } + } + return requireConsumed(cursor, "import"); +} + +[[nodiscard]] Expected parseFunctionSection(Cursor cursor, ModuleInfo* module) { + auto count = cursor.varUint32(); + if (!count) { + return unexpected(count.error()); + } + try { + module->function_types.reserve(module->function_types.size() + *count); + for (uint32_t index = 0; index < *count; ++index) { + auto type_index = cursor.varUint32(); + if (!type_index) { + return unexpected(type_index.error()); + } + module->function_types.push_back(*type_index); + } + } catch (const std::bad_alloc&) { + return unexpected(std::string("allocation failed while reading the wasm function section")); + } + return requireConsumed(cursor, "function"); +} + +[[nodiscard]] Expected parseExportSection(Cursor cursor, ModuleInfo* module) { + auto count = cursor.varUint32(); + if (!count) { + return unexpected(count.error()); + } + try { + module->exports.reserve(*count); + for (uint32_t index = 0; index < *count; ++index) { + auto name = cursor.name(); + auto kind = cursor.byte(); + auto item_index = cursor.varUint32(); + if (!name || !kind || !item_index) { + return unexpected(std::string("truncated wasm export entry")); + } + if (std::any_of( + module->exports.begin(), module->exports.end(), [&](const Export& item) { return item.name == *name; })) { + return unexpected(std::string("duplicate wasm export name: ") + *name); + } + module->exports.push_back(Export{std::move(*name), *kind, *item_index}); + } + } catch (const std::bad_alloc&) { + return unexpected(std::string("allocation failed while reading the wasm export section")); + } + return requireConsumed(cursor, "export"); +} + +[[nodiscard]] Expected inspectModule(Span wasm) { + if (wasm.size() < kWasmPreamble.size() || !std::equal(kWasmPreamble.begin(), kWasmPreamble.end(), wasm.begin())) { + return unexpected(std::string("invalid wasm preamble")); + } + + ModuleInfo module; + Cursor cursor(wasm.subspan(kWasmPreamble.size())); + std::array seen{}; + while (!cursor.empty()) { + auto section_id = cursor.byte(); + auto section_size = cursor.varUint32(); + if (!section_id || !section_size) { + return unexpected(std::string("truncated wasm section header")); + } + if (*section_id > 12) { + return unexpected(std::string("unknown wasm section id")); + } + auto section = cursor.take(*section_size); + if (!section) { + return unexpected(section.error()); + } + ++module.section_count; + if (*section_id != 0) { + if (seen[*section_id]) { + return unexpected(std::string("duplicate standard wasm section")); + } + seen[*section_id] = true; + } + Expected parsed; + switch (*section_id) { + case 1: + parsed = parseTypeSection(*section, &module); + break; + case 2: + parsed = parseImportSection(*section, &module); + break; + case 3: + parsed = parseFunctionSection(*section, &module); + break; + case 7: + parsed = parseExportSection(*section, &module); + break; + case 8: + module.has_start_section = true; + break; + default: + break; + } + if (!parsed) { + return unexpected(parsed.error()); + } + } + if (!seen[1] || !seen[3] || !seen[7]) { + return unexpected(std::string("wasm is missing a type, function, or export section")); + } + for (const uint32_t type_index : module.function_types) { + if (type_index >= module.types.size()) { + return unexpected(std::string("wasm function references an invalid type index")); + } + } + return module; +} + +[[nodiscard]] Expected> readFile(const std::string& path) { + std::ifstream input(path, std::ios::binary | std::ios::ate); + if (!input) { + return unexpected(std::string("cannot open file: ") + path); + } + const std::streamoff end = input.tellg(); + if (end < 0 || static_cast(end) > std::numeric_limits::max() || + end > std::numeric_limits::max()) { + return unexpected(std::string("file size is invalid: ") + path); + } + std::vector bytes(static_cast(end)); + input.seekg(0); + if (!bytes.empty()) { + input.read(reinterpret_cast(bytes.data()), static_cast(bytes.size())); + } + if (!input) { + return unexpected(std::string("cannot read complete file: ") + path); + } + return bytes; +} + +[[nodiscard]] Expected writeFile(const std::string& path, Span bytes) { + std::ofstream output(path, std::ios::binary | std::ios::trunc); + if (!output) { + return unexpected(std::string("cannot create file: ") + path); + } + if (!bytes.empty()) { + output.write(reinterpret_cast(bytes.data()), static_cast(bytes.size())); + } + if (!output) { + return unexpected(std::string("cannot write complete file: ") + path); + } + return {}; +} + +[[nodiscard]] const Export* findExport(const ModuleInfo& module, std::string_view name) { + const Export* result = nullptr; + for (const auto& item : module.exports) { + if (item.name == name) { + if (result != nullptr) { + return nullptr; + } + result = &item; + } + } + return result; +} + +/// One mandatory reactor export and its exact wasm function signature. +struct ExpectedExport { + std::string_view name; + std::vector parameters; + std::vector results; +}; + +[[nodiscard]] Expected requireFunction( + const ModuleInfo& module, std::string_view name, const std::vector& parameters, + const std::vector& results) { + const Export* item = findExport(module, name); + if (item == nullptr) { + return unexpected(std::string("missing or duplicate function export: ") + std::string(name)); + } + if (item->kind != 0 || item->index >= module.function_types.size()) { + return unexpected(std::string("export is not a valid function: ") + std::string(name)); + } + const FunctionType& type = module.types[module.function_types[item->index]]; + if (type.parameters != parameters || type.results != results) { + return unexpected(std::string("function export has the wrong wasm signature: ") + std::string(name)); + } + return {}; +} + +[[nodiscard]] Expected audit(const std::string& wasm_path, const std::string& manifest_path) { + auto wasm = readFile(wasm_path); + auto manifest = readFile(manifest_path); + if (!wasm) { + return unexpected(wasm.error()); + } + if (!manifest) { + return unexpected(manifest.error()); + } + auto embedded = PJ::parser_module::readManifestSection(*wasm); + if (!embedded) { + return unexpected(embedded.error()); + } + if (embedded->size() != manifest->size() || !std::equal(embedded->begin(), embedded->end(), manifest->begin())) { + return unexpected(std::string("embedded parser-module manifest bytes do not match the source file")); + } + + auto module = inspectModule(*wasm); + if (!module) { + return unexpected(module.error()); + } + if (module->has_start_section) { + return unexpected(std::string("wasm reactor contains a forbidden start section")); + } + if (findExport(*module, "_start") != nullptr) { + return unexpected(std::string("wasm reactor exports forbidden _start")); + } + if (findExport(*module, PJ_MODULE_MANIFEST_ADDR_EXPORT_NAME) != nullptr || + findExport(*module, PJ_MODULE_MANIFEST_LEN_EXPORT_NAME) != nullptr) { + return unexpected(std::string("wasm reactor exports native-only manifest metadata")); + } + + const std::array expected{{ + {PJ_MODULE_ABI_EXPORT_NAME, {}, {kI32}}, + {PJ_MODULE_CREATE_EXPORT_NAME, {kI32}, {kI64}}, + {PJ_MODULE_DESTROY_EXPORT_NAME, {kI64}, {}}, + {PJ_MODULE_BIND_EXPORT_NAME, {kI64, kI64, kI64}, {kI32}}, + {PJ_MODULE_PARSE_EXPORT_NAME, {kI64, kI64, kI64, kI64, kI64}, {kI32}}, + {PJ_MODULE_LAST_ERROR_EXPORT_NAME, {kI64, kI64, kI64}, {kI64}}, + {PJ_MODULE_ALLOC_EXPORT_NAME, {kI64}, {kI64}}, + {PJ_MODULE_FREE_EXPORT_NAME, {kI64, kI64}, {}}, + {"_initialize", {}, {}}, + }}; + for (const auto& entry : expected) { + auto valid = requireFunction(*module, entry.name, entry.parameters, entry.results); + if (!valid) { + return unexpected(valid.error()); + } + } + + for (const auto& item : module->exports) { + if (item.name.rfind("pj_module_", 0) != 0) { + continue; + } + const auto found = std::find_if( + expected.begin(), expected.end(), [&](const ExpectedExport& entry) { return entry.name == item.name; }); + if (found == expected.end()) { + return unexpected(std::string("unexpected parser-module export: ") + item.name); + } + } + + std::cout << "WASM parser-module ABI conformance: PASS\n" + << " sections enumerated: " << module->section_count << '\n' + << " function types: " << module->types.size() << ", functions: " << module->function_types.size() + << ", exports: " << module->exports.size() << '\n' + << " operational exports: 8 exact signatures verified\n" + << " reactor: _initialize exported; _start/start section absent\n" + << " native-only metadata exports: absent\n" + << " manifest section: exactly one, " << embedded->size() << " exact bytes\n"; + return {}; +} + +[[nodiscard]] Expected embed( + const std::string& input_path, const std::string& manifest_path, const std::string& output_path) { + auto wasm = readFile(input_path); + auto manifest = readFile(manifest_path); + if (!wasm) { + return unexpected(wasm.error()); + } + if (!manifest) { + return unexpected(manifest.error()); + } + auto output = PJ::parser_module::appendManifestSection(*wasm, *manifest); + if (!output) { + return unexpected(output.error()); + } + return writeFile(output_path, *output); +} + +} // namespace + +int main(int argc, char** argv) { + Expected result = unexpected(std::string("invalid arguments")); + if (argc == 5 && std::string_view(argv[1]) == "--embed") { + result = embed(argv[2], argv[3], argv[4]); + } else if (argc == 4 && std::string_view(argv[1]) == "--audit") { + result = audit(argv[2], argv[3]); + } else { + std::cerr << "usage: parser_module_wasm_audit --embed INPUT.wasm MANIFEST.json OUTPUT.wasm\n" + " or: parser_module_wasm_audit --audit MODULE.wasm MANIFEST.json\n"; + return 2; + } + if (!result) { + std::cerr << "parser-module wasm audit failed: " << result.error() << '\n'; + return 1; + } + return 0; +} diff --git a/pj_base/tests/toy_cdr_pointcloud.module.json b/pj_base/tests/toy_cdr_pointcloud.module.json new file mode 100644 index 00000000..a8b7ebb3 --- /dev/null +++ b/pj_base/tests/toy_cdr_pointcloud.module.json @@ -0,0 +1,24 @@ +{ + "module_abi": 1, + "id": "org.plotjuggler.test.kit-cdr-pointcloud", + "name": "Authoring kit CDR PointCloud fixture", + "version": "1.0.0", + "claims": [ + { + "claim_id": "full-wire", + "encoding": "ros2msg", + "type_name": "toy_msgs/msg/Cloud", + "routes": ["object"], + "object_type": "kPointCloud", + "priority": 0 + }, + { + "claim_id": "spliced", + "encoding": "ros2msg", + "type_name": "toy_msgs/msg/CloudSplice", + "routes": ["object"], + "object_type": "kPointCloud", + "priority": 0 + } + ] +} diff --git a/pj_base/tests/toy_cdr_pointcloud_module.cpp b/pj_base/tests/toy_cdr_pointcloud_module.cpp new file mode 100644 index 00000000..280391b3 --- /dev/null +++ b/pj_base/tests/toy_cdr_pointcloud_module.cpp @@ -0,0 +1,90 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include +#include +#include +#include + +#include "pj_base/parser_module/module.hpp" + +class ToyCdrPointCloudParser : public pj::FunctionalParser { + public: + pj::Status bind(const pj::BindingInfo& info) override { + pj::CdrFieldLocator locator(info.schemaText()); + if (!locator.status().isOk()) { + return pj::Status::decline("unsupported toy schema: " + std::string(locator.status().message())); + } + auto plan = locator.locate({"width", "frame_id", "data"}); + if (!plan) { + return pj::Status::decline("unsupported toy schema revision: " + std::string(plan.status().message())); + } + auto width = plan->field("width"); + auto frame = plan->field("frame_id"); + auto data = plan->field("data"); + if (!width || !frame || !data) { + return pj::Status::decline("unsupported toy schema field plan"); + } + plan_ = std::move(*plan); + width_ = *width; + frame_ = *frame; + data_ = *data; + claim_index_ = info.claimIndex(); + return pj::Status::ok(); + } + + pj::Status parseObject(pj::PayloadView payload, pj::Timestamp timestamp, pj::ObjectWriter& output) override { + pj::CdrReader reader(payload, plan_); + auto width = reader.u32(width_); + auto frame = reader.string(frame_); + auto data = reader.bytes(data_); + if (!width || !frame || !data) { + return pj::Status::error("toy CDR payload does not match the bound schema"); + } + if (*width > std::numeric_limits::max() / 4U) { + return pj::Status::error("toy point-cloud width overflows row_step"); + } + + auto cloud = output.pointCloud(); + if (timestamp.has_value) { + if (auto status = cloud.setTimestamp(timestamp.nanoseconds); !status.isOk()) { + return status; + } + } + if (auto status = cloud.setWidth(*width); !status.isOk()) { + return status; + } + if (auto status = cloud.setHeight(1); !status.isOk()) { + return status; + } + if (auto status = cloud.setPointStep(4); !status.isOk()) { + return status; + } + if (auto status = cloud.setRowStep(*width * 4U); !status.isOk()) { + return status; + } + if (auto status = cloud.setDense(true); !status.isOk()) { + return status; + } + if (auto status = cloud.setFrameId(*frame); !status.isOk()) { + return status; + } + if (claim_index_ == 1) { + auto reference = reader.spanRef(data_); + if (!reference) { + return reference.status(); + } + return cloud.setDataFromInput(*reference); + } + return cloud.setData(*data); + } + + private: + pj::CdrTraversalPlan plan_; + pj::CdrFieldId width_ = 0; + pj::CdrFieldId frame_ = 0; + pj::CdrFieldId data_ = 0; + uint32_t claim_index_ = 0; +}; + +PJ_FUNCTIONAL_PARSER(ToyCdrPointCloudParser) diff --git a/pj_plugins/CLAUDE.md b/pj_plugins/CLAUDE.md index 731a87f6..ea4afcf6 100644 --- a/pj_plugins/CLAUDE.md +++ b/pj_plugins/CLAUDE.md @@ -3,8 +3,9 @@ The runtime-extension layer of `plotjuggler_sdk`: the stable C ABI, the C++ SDK plugin authors subclass, and the host-side loaders/RAII handles that `dlopen` plugin DSOs. Owns **four plugin families** — DataSource, MessageParser, Toolbox, -Dialog. Plugins depend only on `pj_base`; this module (the host side) links -`pj_base` and is consumed by the app. It does **not** own the data-plane bridge +Dialog. The authoring umbrella spans `pj_base` plus the parser/dialog headers in +this module through `plotjuggler_sdk::plugin_sdk`; host libraries link `pj_base` +and are consumed by the app. It does **not** own the data-plane bridge (that is `pj_datastore`'s `DatastoreSourceWriteHost` / `…ParserWriteHost` / `…ToolboxHost`, which now lives in the PlotJuggler application repo, not in this SDK) and links **no Qt** — dialogs are toolkit-neutral (the GUI host supplies the @@ -15,10 +16,13 @@ submodule-internal modules; `pj_base` carries none). ## Layout - `include/pj_plugins/host/` — host loaders + RAII handles for DataSource / MessageParser / Toolbox, the embedded-manifest `plugin_catalog` scanner - (`scanPluginDsos` / `inspectPluginDso`), `ServiceRegistryBuilder`, - `ConfigEnvelope`. The duplicate-resolution catalog that composes these into a - loaded set is **host policy** and lives in the app (`pj_runtime`, - `PluginRuntimeCatalog`), not here. + (`scanPluginDsos` / `inspectPluginDso`), parser claim admission + per-route + resolution (`ParserClaimCatalog`, `ParserRouteResolver`), native functional + parser-module loading/execution (`NativeParserModule`, + `NativeParserModuleInstance`, `ParserModuleStrikeTracker`), + `ServiceRegistryBuilder`, `ConfigEnvelope`. The DSO duplicate-resolution + catalog that composes loaded plugin families into a set is **host policy** and + lives in the app (`pj_runtime`, `PluginRuntimeCatalog`), not here. - `include/pj_plugins/sdk/` — SDK pieces that live here, not in `pj_base`: `MessageParserPluginBase`, `ObjectIngestPolicyResolver`, parser trampolines. - `include/pj_plugins/testing/` — `ToolboxTestStore` (fake Arrow host for tests). @@ -36,13 +40,19 @@ submodule-internal modules; `pj_base` carries none). - **The SDK is split across two modules.** `DataSourcePluginBase` / `ToolboxPluginBase` / `data_source_patterns.hpp` live in **`pj_base/sdk/`**; only `MessageParserPluginBase` + `object_ingest_policy.hpp` live here under - `pj_plugins/sdk/`. (The `docs/ARCHITECTURE.md` §2 diagram is stale on this.) + `pj_plugins/sdk/`. Functional parser-module authoring headers and the native + CMake helper live under `pj_base`; claim resolution and native module + loading/runtime live here. - **Handles keep the DSO mapped.** Every handle holds a `shared_ptr` library token (exposed via `libraryOwner()`), so destroying/hot-reloading the loader cannot `dlclose` a live plugin — and a lazy ObjectStore payload anchor, whose `release` fn is plugin code, can capture that token to stay safe past the handle's own lifetime. Dialog handles add a non-owning `borrowed()` form for source/toolbox embedded dialogs — those must not outlive the owning handle. +- **Native parser modules never unload in v1.** `NativeParserModule` resolves + the complete per-handle export set and retains every opened DSO for the + process session, including rejected artifacts. Instance wrappers still call + `pj_module_destroy`; only the code mapping has session lifetime. ## Read deeper | For | Read | @@ -52,5 +62,8 @@ submodule-internal modules; `pj_base` carries none). | Writing each family | `docs/data-source-guide.md`, `docs/message-parser-guide.md`, `docs/toolbox-guide.md`, `docs/dialog-plugin-guide.md` | | Host loader + factory pattern | `include/pj_plugins/host/data_source_library.hpp`, `…/data_source_handle.hpp` | | Discovery from embedded manifests | `include/pj_plugins/host/plugin_catalog.hpp` (the duplicate-resolution catalog is host-side in `pj_runtime`) | +| Parser claim admission and route selection | `include/pj_plugins/host/parser_claim_catalog.hpp`, `parser_route_resolver.hpp` | +| Native functional parser modules | `include/pj_plugins/host/native_parser_module.hpp`, `parser_module_runtime.hpp` | +| Authoring native functional parser modules | `../pj_base/include/pj_base/parser_module/README.md`, `module.hpp`, `../.claude/skills/plotjuggler-plugin/references/parser-module.md` | | Service wiring into `bind()` | `include/pj_plugins/host/service_registry_builder.hpp` | | Builtin-object ingest policy | `include/pj_plugins/sdk/object_ingest_policy.hpp` | diff --git a/pj_plugins/CMakeLists.txt b/pj_plugins/CMakeLists.txt index bb245f89..2144d4c7 100644 --- a/pj_plugins/CMakeLists.txt +++ b/pj_plugins/CMakeLists.txt @@ -61,6 +61,61 @@ target_link_libraries(pj_plugin_catalog nlohmann_json::nlohmann_json ) +# --------------------------------------------------------------------------- +# pj_parser_claim_catalog — host-side parser claim admission and resolution +# --------------------------------------------------------------------------- + +add_library(pj_parser_claim_catalog STATIC + src/parser_claim_catalog.cpp + src/parser_route_resolver.cpp +) +target_include_directories(pj_parser_claim_catalog PUBLIC + $ + $ +) +target_compile_features(pj_parser_claim_catalog PUBLIC cxx_std_20) +target_compile_options(pj_parser_claim_catalog PRIVATE ${PJ_WARNING_FLAGS}) +set_target_properties(pj_parser_claim_catalog PROPERTIES + POSITION_INDEPENDENT_CODE ON + EXPORT_NAME parser_claim_catalog +) +target_link_libraries(pj_parser_claim_catalog + PUBLIC + pj_base + PRIVATE + nlohmann_json::nlohmann_json +) +add_library(plotjuggler_sdk::parser_claim_catalog ALIAS pj_parser_claim_catalog) + +# --------------------------------------------------------------------------- +# pj_parser_module_host — native functional-module loader and runtime +# --------------------------------------------------------------------------- + +add_library(pj_parser_module_host STATIC + src/native_parser_module.cpp + src/parser_module_runtime.cpp +) +target_include_directories(pj_parser_module_host + PUBLIC + $ + $ + PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/src +) +target_compile_features(pj_parser_module_host PUBLIC cxx_std_20) +target_compile_options(pj_parser_module_host PRIVATE ${PJ_WARNING_FLAGS}) +set_target_properties(pj_parser_module_host PROPERTIES + POSITION_INDEPENDENT_CODE ON + EXPORT_NAME parser_module_host +) +target_link_libraries(pj_parser_module_host + PUBLIC + pj_base + PRIVATE + ${CMAKE_DL_LIBS} +) +add_library(plotjuggler_sdk::parser_module_host ALIAS pj_parser_module_host) + # --------------------------------------------------------------------------- # pj_data_source_host — host-side DataSource library loader # --------------------------------------------------------------------------- @@ -221,8 +276,9 @@ target_link_libraries(pj_toolbox_host # --------------------------------------------------------------------------- # pj_plugin_host — umbrella INTERFACE that bundles every host-side loader. # Hosts (the PlotJuggler app, test rigs) link this single target to get -# discovery + all four plugin-family loaders. The duplicate-resolution catalog -# is host policy and lives in the app (pj_runtime), built on these primitives. +# discovery, all four plugin-family loaders, and functional parser-module +# execution. The duplicate-resolution catalog is host policy and lives in the +# app (pj_runtime), built on these primitives. # --------------------------------------------------------------------------- add_library(pj_plugin_host INTERFACE) @@ -232,12 +288,42 @@ target_link_libraries(pj_plugin_host INTERFACE pj_toolbox_host pj_dialog_library pj_plugin_catalog + pj_parser_claim_catalog + pj_parser_module_host ) set_target_properties(pj_plugin_host PROPERTIES EXPORT_NAME plugin_host) add_library(plotjuggler_sdk::plugin_host ALIAS pj_plugin_host) if(PJ_BUILD_TESTS) +# --------------------------------------------------------------------------- +# Native functional parser-module fixtures (real shared libraries) +# --------------------------------------------------------------------------- + +# Trailing arguments become PRIVATE compile definitions, each selecting one +# deliberately defective variant of the shared fixture source. +function(pj_add_native_parser_module_fixture target) + add_library(${target} SHARED tests/native_parser_module_fixture.cpp) + target_compile_features(${target} PRIVATE cxx_std_20) + target_compile_options(${target} PRIVATE ${PJ_WARNING_FLAGS}) + set_target_properties(${target} PROPERTIES + CXX_VISIBILITY_PRESET hidden + VISIBILITY_INLINES_HIDDEN YES + ) + target_link_libraries(${target} PRIVATE pj_base) + if(ARGN) + target_compile_definitions(${target} PRIVATE ${ARGN}) + endif() + if(UNIX AND NOT APPLE) + target_link_options(${target} PRIVATE "LINKER:-z,defs" "LINKER:--exclude-libs,ALL") + endif() +endfunction() + +pj_add_native_parser_module_fixture(native_parser_module_fixture) +pj_add_native_parser_module_fixture(native_parser_module_missing_export PJ_FIXTURE_OMIT_FREE) +pj_add_native_parser_module_fixture(native_parser_module_wrong_abi PJ_FIXTURE_WRONG_ABI) +pj_add_native_parser_module_fixture(native_parser_module_unreadable_manifest PJ_FIXTURE_UNREADABLE_MANIFEST) + # --------------------------------------------------------------------------- # Mock Toolbox plugin (shared library, for dlopen tests) # --------------------------------------------------------------------------- @@ -327,6 +413,14 @@ target_link_libraries(message_parser_functional_extension_test PRIVATE ) add_test(NAME message_parser_functional_extension_test COMMAND message_parser_functional_extension_test) +# Unit test: exact route claims synthesized from registered schema handlers. +add_executable(message_parser_route_claims_extension_test tests/message_parser_route_claims_extension_test.cpp) +target_compile_options(message_parser_route_claims_extension_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(message_parser_route_claims_extension_test PRIVATE + pj_message_parser_host pj_plugin_sdk GTest::gtest_main +) +add_test(NAME message_parser_route_claims_extension_test COMMAND message_parser_route_claims_extension_test) + # Compile-time pins for the cross-DSO MessageParserPluginBase member layout. add_executable(message_parser_abi_layout_sentinels_test tests/message_parser_abi_layout_sentinels_test.cpp) target_compile_options(message_parser_abi_layout_sentinels_test PRIVATE ${PJ_WARNING_FLAGS}) @@ -344,6 +438,66 @@ target_link_libraries(object_ingest_policy_test PRIVATE target_include_directories(object_ingest_policy_test PRIVATE include) add_test(NAME object_ingest_policy_test COMMAND object_ingest_policy_test) +# Unit tests: parser claim admission, manifest decoding, and synthesis. +add_executable(parser_claim_catalog_test tests/parser_claim_catalog_test.cpp) +target_compile_options(parser_claim_catalog_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(parser_claim_catalog_test PRIVATE + pj_parser_claim_catalog GTest::gtest_main +) +add_test(NAME parser_claim_catalog_test COMMAND parser_claim_catalog_test) + +# Unit tests: deterministic per-route selection and probe-cache invalidation. +add_executable(parser_route_resolver_test tests/parser_route_resolver_test.cpp) +target_compile_options(parser_route_resolver_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(parser_route_resolver_test PRIVATE + pj_parser_claim_catalog GTest::gtest_main +) +add_test(NAME parser_route_resolver_test COMMAND parser_route_resolver_test) + +# Integration tests: native functional parser-module loading and execution. +add_executable(native_parser_module_test tests/native_parser_module_test.cpp) +add_dependencies(native_parser_module_test + native_parser_module_fixture + native_parser_module_missing_export + native_parser_module_wrong_abi + native_parser_module_unreadable_manifest +) +target_compile_definitions(native_parser_module_test PRIVATE + PJ_NATIVE_MODULE_FIXTURE_PATH="$" + PJ_NATIVE_MODULE_MISSING_EXPORT_PATH="$" + PJ_NATIVE_MODULE_WRONG_ABI_PATH="$" + PJ_NATIVE_MODULE_UNREADABLE_MANIFEST_PATH="$" +) +target_compile_options(native_parser_module_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(native_parser_module_test PRIVATE + pj_parser_module_host pj_parser_claim_catalog GTest::gtest_main +) +add_test(NAME native_parser_module_test COMMAND native_parser_module_test) + +add_executable(parser_module_runtime_test tests/parser_module_runtime_test.cpp) +add_dependencies(parser_module_runtime_test native_parser_module_fixture) +target_compile_definitions(parser_module_runtime_test PRIVATE + PJ_NATIVE_MODULE_FIXTURE_PATH="$" +) +target_compile_options(parser_module_runtime_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(parser_module_runtime_test PRIVATE + pj_parser_module_host GTest::gtest_main +) +add_test(NAME parser_module_runtime_test COMMAND parser_module_runtime_test) + +# Keystone integration: a C++17 module authored and built only with the +# header-only kit, then exercised through the production native loader/runtime. +add_executable(parser_module_authoring_e2e_test tests/parser_module_authoring_e2e_test.cpp) +add_dependencies(parser_module_authoring_e2e_test toy_cdr_pointcloud_module) +target_compile_definitions(parser_module_authoring_e2e_test PRIVATE + PJ_TOY_CDR_POINTCLOUD_MODULE_PATH="$" +) +target_compile_options(parser_module_authoring_e2e_test PRIVATE ${PJ_WARNING_FLAGS}) +target_link_libraries(parser_module_authoring_e2e_test PRIVATE + pj_parser_module_host pj_parser_claim_catalog GTest::gtest_main +) +add_test(NAME parser_module_authoring_e2e_test COMMAND parser_module_authoring_e2e_test) + # Unit test: ServiceRegistryBuilder registration rules. PJ_ASSERT_THROWS makes # the convenience overload's invariant observable in every build type — a plain # assert() is compiled away under NDEBUG, which is where it would go unchecked. @@ -435,6 +589,8 @@ if(PJ_INSTALL_SDK) pj_plugin_loader_detail pj_plugin_sdk pj_plugin_catalog + pj_parser_claim_catalog + pj_parser_module_host pj_data_source_host pj_message_parser_host pj_toolbox_host diff --git a/pj_plugins/docs/ARCHITECTURE.md b/pj_plugins/docs/ARCHITECTURE.md index 69548283..053bdb1c 100644 --- a/pj_plugins/docs/ARCHITECTURE.md +++ b/pj_plugins/docs/ARCHITECTURE.md @@ -44,10 +44,11 @@ these is an ABI break and requires a future `PJ_ABI_VERSION` bump. reserved as its one growth path — do not add further top-level fields. - **ABI-APPENDABLE**: all `*_vtable_t` types, service-host vtables, - `PJ_service_registry_vtable_t`, `PJ_dialog_host_info_t`, and the - `pj.parser_functional.v1` extension/sink tables. New fields or slots go at - the tail; readers honor the caller-provided size, and vtable slots are - read with `PJ_HAS_TAIL_SLOT`. + `PJ_service_registry_vtable_t`, `PJ_dialog_host_info_t`, the + `pj.parser_functional.v1`/v2 extension and sink tables, and + `pj.parser_route_claims.v1`. New fields or slots go at the tail; readers + honor the caller-provided size, and vtable slots are read with + `PJ_HAS_TAIL_SLOT`. 5. **Compile-time ABI layout sentinels.** `pj_base/tests/abi_layout_sentinels_test.cpp` consists entirely of `static_assert`s pinning `sizeof`, `alignof`, @@ -114,14 +115,68 @@ plugin instance. C++ wrappers (`DescriptorImportProviderView`, `JoinableJob`, See `docs/toolbox-guide.md` → "Descriptor import and source promotion" for the plugin-author walkthrough. -Stable MessageParser-specific example: `"pj.parser_functional.v1"` -(`PJ_parser_functional_v1_t` in -`pj_base/parser_functional_protocol.h`). SDK 0.21's -`MessageParserPluginBase` exposes it automatically after a plugin registers at -least one `SchemaHandler`. Hosts must query after `bind_schema` and must not -cache an earlier absence because schema-generic plugins may register their -handler while binding. A parser that still implements only legacy `parse()` -does not advertise the extension merely because it was rebuilt with 0.21. +Stable MessageParser-specific examples are `"pj.parser_functional.v1"` and +`"pj.parser_functional.v2"` (`PJ_parser_functional_v1_t` and +`PJ_parser_functional_v2_t` in `pj_base/parser_functional_protocol.h`) plus the +exact route classifier `"pj.parser_route_claims.v1"`. A handler-registering +`MessageParserPluginBase` exposes all three automatically. The host queries v2 +first and falls back to v1; v2 adds one eligible object-field splice while +leaving scalar parsing unchanged. Route classification reports exact handler +coverage only, and the host synthesizes the universal wildcard scalar claim. +Hosts must query after `bind_schema` and must not cache an earlier absence +because schema-generic plugins may register their handler while binding. A +parser that still implements only legacy `parse()` does not advertise the +functional extensions merely because it was rebuilt. + +### Functional parser modules + +Functional parser modules use the family-independent export ABI in +`pj_base/parser_module_abi.h`, not a plugin-family vtable. The host-side +`NativeParserModule` loader opens each artifact with local, immediate symbol +resolution, resolves every operational and native metadata export, gates ABI +version 1, and copies the embedded manifest for subsequent +`ParserClaimCatalog` admission. Native module mappings have process-session +lifetime in v1 and are never unloaded. + +`NativeParserModuleInstance` owns one create/bind/parse/destroy lifecycle. It +uses the frozen little-endian codecs for input and output, consumes all +module-owned descriptor views before returning, validates output route and +expected object type, and checks a splice against both the canonical object's +eligible field and the original payload bounds. A valid splice is materialized +into that field, so the returned canonical object is complete; its sidecar +retains the original offset and bytes for future zero-copy integration. +Malformed returned descriptors, ineligible/out-of-bounds returned splices, +type/route mismatch, and bad tokens are contract violations; other +module-reported per-message failures are data errors. The separate +non-thread-safe `ParserModuleStrikeTracker` quarantines a module claim on its +third contract violation, permits one same-descriptor recreation, and disables +the claim after a repeated three-strike cycle. Executor placement, generation +ownership, folder scanning, and rescan policy remain application concerns. + +Module authors use the standalone C++17 headers under +`pj_base/include/pj_base/parser_module/` through the zero-linkage +`plotjuggler_sdk::parser_module` target. The readers compile ROS 2 concatenated +`.msg` bundles or protobuf `FileDescriptorSet` field paths at bind time; the +CDR locator supports fixed arrays, bounded and unbounded sequences, bounded +strings, and string arrays/sequences, while cyclic or over-depth schemas fail at +bind. The typed `ObjectWriter` emits Image, PointCloud, DepthImage, +OccupancyGrid, CompressedPointCloud, Mesh3D, VideoFrame, OccupancyGridUpdate, +and VoxelGrid descriptors, including each type's eligible single-splice form. +Module parse callbacks receive the per-message `pj::Timestamp` alongside the +payload. `BindingInfo` views expire when `bind()` returns; `owningCopy()` is the +fallible retention path. Bulk storage uses nothrow allocation, and protobuf +matching is bounded, so allocation failure is returned as data error rather +than escaping as a trap. `PJ_FUNCTIONAL_PARSER` supplies the complete native +export set, uses synchronized index+generation instance tokens to reject stale +handles, and catches user exceptions at the C boundary. +`pj_add_parser_module(... TARGETS native)` builds a hidden-visibility module and +embeds its JSON manifest behind the native metadata exports. `TARGETS wasm` is +not available in SDK 0.22. The shared host codec can append and read the exact +JSON bytes in a wasm `pj_parser_module_manifest` custom section; the conditional +wasi-sdk 27 compile gate builds the toy source with C++17 and exceptions +disabled, then statically audits the reactor model and every operational export +signature. This is structural conformance only: no wasm loader or execution +runtime ships in this release. ## 0. C protocol v4 (current under ABI v5) @@ -230,7 +285,7 @@ Every plugin family follows the same three-level pattern: ``` C ABI protocol → C++ SDK base class → Host loader + RAII handle - (pj_base) (pj_base) (pj_plugins) + (pj_base) (pj_base/pj_plugins) (pj_plugins) ``` 1. **C ABI protocol** — a vtable struct in a plain-C header. Defines the @@ -253,7 +308,17 @@ pj_base/ include/pj_base/ data_source_protocol.h ← C ABI message_parser_protocol.h ← C ABI - parser_functional_protocol.h ← C ABI: pj.parser_functional.v1 sinks + parser_functional_protocol.h ← C ABI: pj.parser_functional.v1/v2 sinks + parser_route_claims_protocol.h ← C ABI: exact parser route classification + parser_module_abi.h ← native/wasm module exports + byte codecs + parser_module_manifest.hpp ← wasm manifest custom-section embed/read codec + parser_module/ ← standalone C++17 header-only module authoring kit + module.hpp ← umbrella API + PJ_FUNCTIONAL_PARSER exports + cdr_reader.hpp ← bounded XCDR1 reader + cdr_field_locator.hpp ← ROS 2 .msg field-path compiler/cache + proto_reader.hpp ← bounded protobuf wire reader + proto_field_locator.hpp ← FileDescriptorSet field-path compiler + object_writer.hpp ← nine splice-eligible canonical object builders toolbox_protocol.h ← C ABI plugin_data_api.h ← shared data-plane ABI (write hosts) descriptor_import_protocol.h ← C ABI: pj.descriptor_import.v1 extension + @@ -291,6 +356,10 @@ pj_plugins/ toolbox_library.hpp toolbox_handle.hpp plugin_catalog.hpp ← embedded-manifest DSO scanner (scanPluginDsos / inspectPluginDso) + parser_claim_catalog.hpp ← parser claims, manifest admission, plugin-claim synthesis + parser_route_resolver.hpp ← ordered per-route selection + probe cache + native_parser_module.hpp ← session-lifetime native module loader + parser_module_runtime.hpp ← module instances, outputs, fault tracking service_registry_builder.hpp ← service wiring into bind() config_envelope.hpp ← versioned config wrapper include/pj_plugins/sdk/ @@ -303,8 +372,15 @@ pj_plugins/ src/ data_source_library.cpp message_parser_library.cpp + parser_claim_catalog.cpp + parser_route_resolver.cpp + native_parser_module.cpp + parser_module_runtime.cpp toolbox_library.cpp +cmake/ + PjParserModule.cmake ← pj_add_parser_module native target helper + (PlotJuggler application repo — not part of this SDK submodule) pj_datastore/ include/pj_datastore/ @@ -313,9 +389,12 @@ pj_datastore/ DatastoreToolboxHost ``` -**Dependency direction:** Plugins depend only on `pj_base`. The host links -`pj_plugins` (which depends on `pj_base`). `pj_datastore` — now a module in the -PlotJuggler application repo, not part of this SDK — provides the concrete +**Dependency direction:** Installed plugin authors consume +`plotjuggler_sdk::plugin_sdk`, which combines `pj_base` with the MessageParser +and dialog authoring headers from `pj_plugins`. Functional parser modules depend +only on the zero-linkage `plotjuggler_sdk::parser_module` header target. Host +libraries in `pj_plugins` depend on `pj_base`. `pj_datastore` — now a module in +the PlotJuggler application repo, not part of this SDK — provides the concrete data-host implementations that bridge plugin writes to the columnar storage engine. @@ -404,6 +483,23 @@ adapter needed to marshal diagnostics onto their UI thread. A default-constructed sink discards events at zero cost, so loaders that take no sink behave as before. +### 5.2 Parser claim catalog and route resolver + +`ParserClaimCatalog` is the host-side parser dispatch catalog. It admits +transactional claim batches from module manifests or synthesized parser-plugin +coverage, validates stable `(provider_id, claim_id)` identities, and attaches +host-supplied provenance and provider generations. The SDK-owned encoding +registry and normalization helpers keep matching case-sensitive and canonical. + +`ParserRouteResolver` independently selects scalar and object providers. It +applies fail-closed per-route pins, exact-before-wildcard specificity, +provenance, bounded priority, and stable identity ordering before invoking a +caller-supplied probe. Probe decisions and retained opaque leases are cached by +provider generation plus binding identity/config digests. Catalog, pin, and +provider-config mutation paths explicitly invalidate that cache. The resolver +contains no loader or executor: the embedding host runs the callback on its +parser-control executor and owns the concrete provider instance type. + ## 6. RAII Handles Each family has a move-only RAII handle: @@ -629,8 +725,8 @@ that cascades `topic > source > type > default`: Parsers participate through `classifySchema`, `parseScalars`, and `parseObject`, backed by the per-schema `SchemaHandler` table. Those C++ methods and their `ScalarRecord` / `ObjectRecord` / `std::any` values stay -inside the plugin DSO. SDK 0.21 translates the two functional results through -`pj.parser_functional.v1`: +inside the plugin DSO. The base exposes both `pj.parser_functional.v1` and v2; +`MessageParserHandle` negotiates v2 first and falls back to v1: - `parse_scalars` calls one caller-owned sink exactly once. Field/name/string views are borrowed only for that callback; the host copies anything it @@ -639,11 +735,14 @@ inside the plugin DSO. SDK 0.21 translates the two functional results through and releasing it on every path. That preserves the existing zero-copy input into parsers that propagate `PayloadView::anchor`. It serializes the plugin's concrete builtin to its canonical `PJ.*` protobuf wire contract and calls one - object sink with `(BuiltinObjectType, optional timestamp, bytes)`. + object sink with `(BuiltinObjectType, optional timestamp, bytes)`. The v2 sink + may instead carry one eligible bulk field as a payload-relative splice. - `MessageParserHandle::parseObjectFunctional` decodes those bytes before the - callback returns, producing an entirely host-owned `ObjectRecord`. Its - destructor and `std::any` manager contain no plugin function pointer, so the - value remains safe after the parser and DSO lease are destroyed. + callback returns, reconstructs a v2 splice into the canonical object, and + rejects a type different from the binding's expected object type. The result + is an entirely host-owned `ObjectRecord`; its destructor and `std::any` + manager contain no plugin function pointer, so the value remains safe after + the parser and DSO lease are destroyed. - Provider exceptions, consumer exceptions, malformed/undersized sinks, unknown object tags, missing calls, and duplicate calls fail closed through `PJ_error_t`. The built-in extension table and trampolines have DSO-local @@ -665,10 +764,11 @@ zero-length canonical payload is accepted as the valid proto3 default message when the separate type tag is known. Concrete builtins and codecs live under `pj_base/builtin/`; see `docs/builtin_type.md` for the catalog. -Pre-0.21 parsers expose no functional extension. PlotJuggler 0.21 may retain a -clearly isolated, deprecated direct-C++ bridge for those already-built DSOs. -New plugins and new host code use the C extension; SDK 1.0 can remove the bridge -and the frozen `MessageParserPluginBase` layout constraint. +Pre-0.21 parsers expose no functional extension, and handler-based 0.21 parsers +may expose v1 without v2. The host keeps a clearly isolated, deprecated +direct-C++ bridge for binaries with neither extension. New plugins and new host +code use v2-first negotiation; SDK 1.0 can remove the bridge and the frozen +`MessageParserPluginBase` layout constraint. ## Per-topic pause (demand-driven subscription) diff --git a/pj_plugins/docs/REQUIREMENTS.md b/pj_plugins/docs/REQUIREMENTS.md index 85be7e27..05d3a8fa 100644 --- a/pj_plugins/docs/REQUIREMENTS.md +++ b/pj_plugins/docs/REQUIREMENTS.md @@ -29,8 +29,10 @@ Four plugin families exist: virtuals, and export with a macro. The SDK generates C ABI trampolines with full exception safety. -3. **No Qt dependency in the plugin SDK.** Plugins link only `pj_base` (and - optionally `pj_dialog_sdk` for dialog UI). Qt is a host-side concern. +3. **No Qt dependency in the plugin SDK.** Installed plugins link the + `plotjuggler_sdk::plugin_sdk` umbrella, which supplies `pj_base` plus the + parser/dialog authoring headers without linking Qt. Qt is a host-side + concern. 4. **Version negotiation.** Each protocol carries `protocol_version` and `struct_size` for forward/backward compatibility. @@ -101,9 +103,15 @@ Independent decoder, typically driven by a DataSource via the host. JSON. - Lifecycle: create → bind → parse* → destroy. - A parser that registers `SchemaHandler` entries exposes - `pj.parser_functional.v1` after schema binding. Functional scalar and object - results cross only through synchronous caller-owned C sinks; the host must - never cast the opaque context to `MessageParserPluginBase` for such a parser. + `pj.parser_functional.v1`, `pj.parser_functional.v2`, and + `pj.parser_route_claims.v1` after schema binding. Functional scalar and object + results cross only through synchronous caller-owned C sinks; v2 may represent + one eligible object bulk field as a payload-relative splice. The host queries + v2 first, falls back to v1, and must never cast the opaque context to + `MessageParserPluginBase` for such a parser. +- Route classification reports exact handler-table claims only. The host owns + each encoding's universal wildcard scalar claim and resolves scalar and object + routes independently. - Scalar views are callback-duration-only. Canonical objects cross as a stable numeric builtin tag plus canonical wire bytes and are decoded into host-owned storage before the extension call returns. No plugin allocator, destructor, @@ -112,13 +120,13 @@ Independent decoder, typically driven by a DataSource via the host. releases it exactly once on success and every failure path. This permits zero-copy input when the host already owns an anchored payload; output still pays the explicit canonical serialization boundary. -- Extension presence is truthful, not merely SDK-version-based: a 0.21-built +- Extension presence is truthful, not merely SDK-version-based: a rebuilt parser that still overrides only legacy `parse()` must not advertise the - functional extension. Hosts query after `bind_schema` and do not cache an - earlier absence. -- During the 0.21 migration only, a host may use an isolated deprecated - direct-C++ bridge for a pre-0.21 parser whose extension is absent. The bridge - is removed, together with its class-layout constraint, in SDK 1.0. + functional extensions. Hosts query after `bind_schema` and do not cache an + earlier absence; an older handler parser may expose v1 without v2. +- Until SDK 1.0, a host may use an isolated deprecated direct-C++ bridge for a + pre-0.21 parser whose functional extensions are absent. The bridge and its + class-layout constraint are removed together at 1.0. ### Dialog diff --git a/pj_plugins/docs/message-parser-guide.md b/pj_plugins/docs/message-parser-guide.md index d0d06242..7fa761d0 100644 --- a/pj_plugins/docs/message-parser-guide.md +++ b/pj_plugins/docs/message-parser-guide.md @@ -3,9 +3,9 @@ > **Tracks the v5 plugin ABI** (`PJ_ABI_VERSION == 5`). The parser > write-host supports per-record writes and an optional Arrow stream > batch path for parser-shaped formats that naturally decode batches. -> SchemaHandler-based parsers additionally expose the -> `pj.parser_functional.v1` C extension introduced in SDK 0.21; no C++ result -> object crosses the plugin boundary. +> SchemaHandler-based parsers additionally expose +> `pj.parser_functional.v1`, `pj.parser_functional.v2`, and +> `pj.parser_route_claims.v1`; no C++ result object crosses the plugin boundary. > For ABI evolution rules, error semantics, and noexcept discipline see > `ARCHITECTURE.md`. @@ -26,6 +26,11 @@ already selected a topic/encoding and is handing you payloads to decode. If one `parse()` call naturally yields a batch, the parser write host can accept that batch via `writeHost().appendArrowStream(...)`. +If a supported encoding already exists and you only need one or a few exact +custom message types rendered as canonical objects or scalars, use a native +**functional parser module** instead of owning or forking the whole encoding. +See `.claude/skills/plotjuggler-plugin/references/parser-module.md`. + A MessageParser is the right choice when: - Each message is independently decodable (JSON line, single Protobuf message, ROS message, Influx line). @@ -39,9 +44,9 @@ A MessageParser is the right choice when: A MessageParser plugin is a shared library (`.so` / `.dylib` / `.dll`) that decodes raw byte payloads — JSON, Protobuf, ROS messages, Influx line protocol, -etc. — into named numeric fields that PlotJuggler can plot. Plugins link only -against `pj_base` (no Qt, no host internals) and communicate through a stable -C ABI. +etc. — into named numeric fields that PlotJuggler can plot. Plugins link against +the `plotjuggler_sdk::plugin_sdk` authoring target (no Qt or host internals) and +communicate through a stable C ABI. MessageParsers are typically used via **delegated ingest**: a DataSource plugin acquires raw data from files or network streams and pushes raw payloads through @@ -55,7 +60,7 @@ the host, which routes them to the appropriate parser based on encoding name. `parse()` instead. Optionally override `bindSchema()`, `saveConfig()`, and `loadConfig()`. 3. Export with `PJ_MESSAGE_PARSER_PLUGIN(YourClass, R"({"id":"...","name":"...","version":"...","encoding":["..."]})")` -4. Build as a shared library linking `pj_base` +4. Build as a shared library linking `plotjuggler_sdk::plugin_sdk` For a namespaced class in a static build, provide the getter-symbol token separately: `PJ_MESSAGE_PARSER_PLUGIN_NAMED(my::Parser, MyParser, kManifest)`. @@ -64,7 +69,7 @@ Dynamic builds may use either form. A complete example lives at `pj_plugins/examples/mock_json_parser.cpp`. `parse()` remains supported for scalar push ingestion, but it does not by -itself advertise functional parsing. A 0.21 rebuild of a legacy parser therefore +itself advertise functional parsing. A rebuild of a legacy parser therefore stays on the host's compatibility path until it registers a `SchemaHandler`; there is no false capability claim and no flag to maintain manually. @@ -100,7 +105,7 @@ trampolines; others prevent runtime parse failures. ### 1. Declare your class ```cpp -#include +#include class MyJsonParser : public PJ::MessageParserPluginBase { public: @@ -158,15 +163,13 @@ can read it without creating an instance. ```cmake add_library(my_parser_plugin SHARED my_parser.cpp) -target_link_libraries(my_parser_plugin PRIVATE pj_base) +target_link_libraries(my_parser_plugin PRIVATE plotjuggler_sdk::plugin_sdk) ``` No other dependencies are needed for headless parsers. If your parser includes -a configuration dialog (see Dialog Integration below), also link `pj_dialog_sdk`: - -```cmake -target_link_libraries(my_parser_plugin PRIVATE pj_base pj_dialog_sdk) -``` +a configuration dialog, the same installed umbrella target includes the dialog +SDK. (`pj_base` and `pj_dialog_sdk` are in-tree implementation targets, not the +downstream authoring interface.) ## Lifecycle @@ -292,6 +295,24 @@ if (!status) { ## Optional Features +### Route claims and functional v1/v2 + +Registering at least one `SchemaHandler` automatically exposes three extension +tables; parser authors do not implement them by hand: + +- `pj.parser_route_claims.v1` classifies exact handler-table coverage. It never + reports wildcard claims. The host synthesizes the universal wildcard scalar + claim from each manifest encoding. +- `pj.parser_functional.v1` delivers scalar records and complete canonical-wire + objects through synchronous caller-owned sinks. +- `pj.parser_functional.v2` keeps the scalar contract and adds a sink for one + splice-eligible object field. `MessageParserHandle` queries v2 first and falls + back to v1 when it is absent. + +The host resolves scalar and object routes independently. A parser may keep the +scalar route for a type it flattens while another provider owns that type's +canonical-object route. + ### Schema binding Override `bindSchema()` to receive schema data before parsing begins. The @@ -309,7 +330,8 @@ PJ::Status bindSchema(std::string_view type_name, ``` The `type_name` is the encoding-specific message type (e.g. -`"sensor_msgs/Imu"` for ROS, `"my.package.ImuSample"` for Protobuf). The +`"sensor_msgs/msg/Imu"` for `ros2msg`, `"my.package.ImuSample"` for +Protobuf). The `schema` bytes are encoding-specific (e.g. ROS `.msg` definition text, Protobuf `FileDescriptorSet` binary). @@ -448,7 +470,7 @@ it without instantiating the plugin. | `id` | string | yes | Stable plugin identifier used by the host catalog. Must be unique per plugin. | | `name` | string | yes | Human-readable plugin name. | | `version` | string | yes | Semver version string. | -| `encoding` | array of strings | yes | Encodings this parser handles, e.g. `["json"]`, `["protobuf"]`, `["ros1msg", "ros2msg", "cdr"]`. The host uses this list to match binding requests to parsers; one parser may register more than one encoding. | +| `encoding` | array of strings | yes | Case-sensitive registered encodings this parser handles, e.g. `["json"]`, `["protobuf"]`, or `["ros1msg", "ros2msg"]`. The host uses this list to match binding requests to parsers; one parser may register more than one encoding. | Example: ```json @@ -634,21 +656,23 @@ handler.parse_object = `std::any_cast(&obj)` to dispatch to the matching viewer. The types above are plugin-author C++ conveniences, not the binary boundary. -After `bindSchema`, `MessageParserPluginBase` advertises -`pj.parser_functional.v1` when at least one handler exists. Its trampoline: +After `bindSchema`, `MessageParserPluginBase` advertises functional v1 and v2 +when at least one handler exists. Their trampolines: 1. invokes `parseScalars` / `parseObject` inside the plugin DSO; 2. delivers scalar fields through one synchronous caller-owned C sink, or - serializes the builtin to canonical `PJ.*` protobuf wire bytes and delivers - `(type, optional timestamp, bytes)` through one object sink; + serialize the builtin to canonical `PJ.*` protobuf wire bytes and deliver + `(type, optional timestamp, bytes)` through one object sink; v2 may instead + identify one frozen bulk field as a splice into the parse payload; 3. destroys every plugin-side C++ result before returning. -`MessageParserHandle::parseScalarsFunctional()` copies or consumes scalar -views only during the callback. `parseObjectFunctional()` decodes canonical -bytes synchronously into a host-owned `ObjectRecord`; that value remains safe -after both the parser instance and shared library are gone. Provider/sink -exceptions, unknown object tags, malformed tables, and zero/multiple sink calls -fail closed. +`MessageParserHandle::parseScalarsFunctional()` copies or consumes scalar views +only during the callback. `parseObjectFunctional()` negotiates v2 first, +reconstructs a valid splice when present, validates that the emitted type equals +the binding's expected object type, and decodes synchronously into a host-owned +`ObjectRecord`; that value remains safe after both the parser instance and +shared library are gone. Provider/sink exceptions, unknown or mismatched object +tags, malformed tables or splices, and zero/multiple sink calls fail closed. For large objects, pass the `sdk::PayloadView` overload rather than a bare `Span` when an ownership anchor already exists. The C extension transfers one @@ -660,9 +684,11 @@ destructors, RTTI, STL layout, or `std::any` manager functions. Measure this path for image/point-cloud workloads instead of bypassing it with direct C++ calls. -Parsers built before SDK 0.21 expose no functional extension. PlotJuggler 0.21 -may use a deprecated, isolated direct-C++ bridge only for those binaries. New -host code must not cast an opaque parser context; SDK 1.0 removes the bridge. +Parsers built before SDK 0.21 expose no functional extension, and handler-based +0.21 parsers may expose v1 without v2. New host code uses v2-first/v1-fallback +negotiation and may use a deprecated, isolated direct-C++ bridge only when both +extensions are absent. It must not otherwise cast an opaque parser context; SDK +1.0 removes the bridge. Builtin types and their canonical codecs live under `pj_base/builtin/`, one header per type. `serializeBuiltinObject()` / `deserializeBuiltinObject()` diff --git a/pj_plugins/include/pj_plugins/host/message_parser_handle.hpp b/pj_plugins/include/pj_plugins/host/message_parser_handle.hpp index bba46bb6..a492b447 100644 --- a/pj_plugins/include/pj_plugins/host/message_parser_handle.hpp +++ b/pj_plugins/include/pj_plugins/host/message_parser_handle.hpp @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -25,6 +26,7 @@ #include #include #include +#include namespace PJ { @@ -48,7 +50,10 @@ class MessageParserHandle { } MessageParserHandle(MessageParserHandle&& other) noexcept - : vt_(other.vt_), ctx_(other.ctx_), library_owner_(std::move(other.library_owner_)) { + : vt_(other.vt_), + ctx_(other.ctx_), + library_owner_(std::move(other.library_owner_)), + expected_object_type_(other.expected_object_type_) { other.vt_ = nullptr; other.ctx_ = nullptr; } @@ -58,6 +63,7 @@ class MessageParserHandle { std::swap(vt_, other.vt_); std::swap(ctx_, other.ctx_); std::swap(library_owner_, other.library_owner_); + std::swap(expected_object_type_, other.expected_object_type_); } return *this; } @@ -82,12 +88,20 @@ class MessageParserHandle { } [[nodiscard]] Status bindSchema(std::string_view type_name, Span schema) { + expected_object_type_.reset(); PJ_string_view_t tn{type_name.data(), type_name.size()}; PJ_bytes_view_t sc{schema.data(), schema.size()}; PJ_error_t err{}; if (!vt_->bind_schema(ctx_, tn, sc, &err)) { return unexpected(errorToString(err)); } + if (PJ_HAS_TAIL_SLOT(PJ_message_parser_vtable_t, vt_, classify_schema)) { + PJ_schema_classification_t classification{}; + err = {}; + if (vt_->classify_schema(ctx_, tn, sc, &classification, &err)) { + expected_object_type_ = static_cast(classification.object_type); + } + } return okStatus(); } @@ -123,16 +137,19 @@ class MessageParserHandle { /// Older parsers return false and can be handled through the deprecated /// in-process C++ compatibility bridge until the next SDK major version. [[nodiscard]] bool supportsFunctionalParsing() const { - return functionalExtension() != nullptr; + const auto extensions = functionalExtensions(); + return extensions.v2 != nullptr || extensions.v1 != nullptr; } /// Parse one scalar record and synchronously deliver borrowed C ABI values /// to @p sink. The sink must copy names/string values it retains. [[nodiscard]] Status parseScalarsFunctional( Timestamp timestamp_ns, Span payload, const ScalarRecordSink& sink) const { - const auto* extension = functionalExtension(); - if (extension == nullptr) { - return unexpected(std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); + const auto extensions = functionalExtensions(); + if (extensions.v2 == nullptr && extensions.v1 == nullptr) { + return unexpected( + std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V2 + " or " + + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); } if (!sink) { return unexpected(std::string("scalar record sink is empty")); @@ -146,7 +163,8 @@ class MessageParserHandle { }; PJ_error_t error{}; const PJ_bytes_view_t bytes{payload.data(), payload.size()}; - if (!extension->parse_scalars(ctx_, timestamp_ns, bytes, &abi_sink, &error)) { + const auto parse_scalars = extensions.v2 != nullptr ? extensions.v2->parse_scalars : extensions.v1->parse_scalars; + if (!parse_scalars(ctx_, timestamp_ns, bytes, &abi_sink, &error)) { return unexpected(sdk::errorToString(error)); } if (state.call_count != 1) { @@ -159,12 +177,15 @@ class MessageParserHandle { /// are decoded synchronously into host-owned SDK storage before returning. [[nodiscard]] Expected parseObjectFunctional( Timestamp timestamp_ns, Span payload) const { - const auto* extension = functionalExtension(); - if (extension == nullptr) { - return unexpected(std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); + const auto extensions = functionalExtensions(); + if (extensions.v2 == nullptr && extensions.v1 == nullptr) { + return unexpected( + std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V2 + " or " + + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); } return parseObjectFunctionalAbi( - extension, timestamp_ns, PJ_payload_t{.data = payload.data(), .size = payload.size(), .anchor = {}}); + extensions, timestamp_ns, PJ_payload_t{.data = payload.data(), .size = payload.size(), .anchor = {}}, payload, + {}); } /// Anchored overload for zero-copy input into the plugin. Ownership of one @@ -172,18 +193,21 @@ class MessageParserHandle { /// plugin's object has been serialized, including every failure path. [[nodiscard]] Expected parseObjectFunctional( Timestamp timestamp_ns, sdk::PayloadView payload) const { - const auto* extension = functionalExtension(); - if (extension == nullptr) { - return unexpected(std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); + const auto extensions = functionalExtensions(); + if (extensions.v2 == nullptr && extensions.v1 == nullptr) { + return unexpected( + std::string("message parser does not expose ") + PJ_PARSER_FUNCTIONAL_EXTENSION_V2 + " or " + + PJ_PARSER_FUNCTIONAL_EXTENSION_V1); } PJ_payload_t abi_payload{.data = payload.bytes.data(), .size = payload.bytes.size(), .anchor = {}}; + sdk::BufferAnchor splice_anchor = payload.anchor; if (payload.anchor) { auto* held = new sdk::BufferAnchor(std::move(payload.anchor)); abi_payload.anchor.ctx = held; abi_payload.anchor.release = [](void* ctx) noexcept { delete static_cast(ctx); }; } - return parseObjectFunctionalAbi(extension, timestamp_ns, abi_payload); + return parseObjectFunctionalAbi(extensions, timestamp_ns, abi_payload, payload.bytes, std::move(splice_anchor)); } /// A priori classification of the bound schema. Tail-slot gated; when @@ -201,7 +225,9 @@ class MessageParserHandle { if (!vt_->classify_schema(ctx_, tn, sc, &out, &err)) { return sdk::BuiltinObjectType::kNone; } - return static_cast(out.object_type); + const auto result = static_cast(out.object_type); + expected_object_type_ = result; + return result; } /// Query a plugin-exposed extension by reverse-DNS id. Tail-slot gated. @@ -222,17 +248,41 @@ class MessageParserHandle { } private: + struct FunctionalExtensions { + const PJ_parser_functional_v2_t* v2 = nullptr; + const PJ_parser_functional_v1_t* v1 = nullptr; + }; + [[nodiscard]] Expected parseObjectFunctionalAbi( - const PJ_parser_functional_v1_t* extension, Timestamp timestamp_ns, PJ_payload_t payload) const { - ObjectSinkState state; - PJ_parser_object_sink_v1_t abi_sink{ - .struct_size = sizeof(PJ_parser_object_sink_v1_t), - .ctx = &state, - .accept_object = acceptObject, + FunctionalExtensions extensions, Timestamp timestamp_ns, PJ_payload_t payload, Span input_payload, + sdk::BufferAnchor input_anchor) const { + ObjectSinkState state{ + .record = {}, + .input_payload = input_payload, + .input_anchor = std::move(input_anchor), + .expected_object_type = expected_object_type_, + .call_count = 0, }; PJ_error_t error{}; - if (!extension->parse_object(ctx_, timestamp_ns, payload, &abi_sink, &error)) { - return unexpected(sdk::errorToString(error)); + if (extensions.v2 != nullptr) { + PJ_parser_object_sink_v2_t abi_sink{ + .struct_size = sizeof(PJ_parser_object_sink_v2_t), + .ctx = &state, + .accept_object = acceptObject, + .accept_object_spliced = acceptObjectSpliced, + }; + if (!extensions.v2->parse_object(ctx_, timestamp_ns, payload, &abi_sink, &error)) { + return unexpected(sdk::errorToString(error)); + } + } else { + PJ_parser_object_sink_v1_t abi_sink{ + .struct_size = sizeof(PJ_parser_object_sink_v1_t), + .ctx = &state, + .accept_object = acceptObject, + }; + if (!extensions.v1->parse_object(ctx_, timestamp_ns, payload, &abi_sink, &error)) { + return unexpected(sdk::errorToString(error)); + } } if (state.call_count != 1 || !state.record.has_value()) { return unexpected(std::string("functional parser returned success without exactly one canonical object")); @@ -247,17 +297,26 @@ class MessageParserHandle { struct ObjectSinkState { std::optional record; + Span input_payload; + sdk::BufferAnchor input_anchor; + std::optional expected_object_type; uint32_t call_count = 0; }; - [[nodiscard]] const PJ_parser_functional_v1_t* functionalExtension() const { - const auto* extension = + [[nodiscard]] FunctionalExtensions functionalExtensions() const { + const auto* v2 = + static_cast(getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2)); + if (v2 == nullptr || v2->struct_size < PJ_PARSER_FUNCTIONAL_V2_MIN_SIZE || v2->parse_scalars == nullptr || + v2->parse_object == nullptr) { + v2 = nullptr; + } + const auto* v1 = static_cast(getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V1)); - if (extension == nullptr || extension->struct_size < PJ_PARSER_FUNCTIONAL_V1_MIN_SIZE || - extension->parse_scalars == nullptr || extension->parse_object == nullptr) { - return nullptr; + if (v1 == nullptr || v1->struct_size < PJ_PARSER_FUNCTIONAL_V1_MIN_SIZE || v1->parse_scalars == nullptr || + v1->parse_object == nullptr) { + v1 = nullptr; } - return extension; + return {.v2 = v2, .v1 = v1}; } static bool acceptScalarRecord( @@ -308,14 +367,22 @@ class MessageParserHandle { return false; } if (canonical_wire.data == nullptr && canonical_wire.size != 0) { - sdk::fillError(out_error, 2, "host", "object callback received invalid canonical wire bytes"); + fillContractViolation(out_error, "object callback received invalid canonical wire bytes"); + return false; + } + if (!validateExpectedObjectType(*state, object_type, out_error)) { return false; } try { + if (canonical_wire.size > std::numeric_limits::max()) { + fillContractViolation(out_error, "canonical object wire exceeds the host size range"); + return false; + } auto object = deserializeBuiltinObject( - static_cast(object_type), canonical_wire.data, canonical_wire.size); + static_cast(object_type), canonical_wire.data, + static_cast(canonical_wire.size)); if (!object) { - sdk::fillError(out_error, 1, "host", std::move(object).error()); + fillContractViolation(out_error, std::move(object).error()); return false; } state->record = sdk::ObjectRecord{ @@ -332,9 +399,142 @@ class MessageParserHandle { } } + static bool acceptObjectSpliced( + void* ctx, bool has_timestamp, int64_t timestamp_ns, uint16_t object_type, PJ_bytes_view_t canonical_wire, + uint32_t splice_field_number, uint64_t input_offset, uint64_t input_length, PJ_error_t* out_error) noexcept { + auto* state = static_cast(ctx); + if (state == nullptr) { + fillContractViolation(out_error, "spliced-object callback received invalid context"); + return false; + } + ++state->call_count; + if (state->call_count != 1) { + fillContractViolation(out_error, "functional parser emitted more than one canonical object"); + return false; + } + if (!validateExpectedObjectType(*state, object_type, out_error)) { + return false; + } + uint32_t eligible_field = 0; + if (!pj_builtin_object_splice_field_number_v1(object_type, &eligible_field) || + splice_field_number != eligible_field) { + fillContractViolation(out_error, "functional parser emitted an ineligible object splice field"); + return false; + } + const uint64_t payload_size = state->input_payload.size(); + if (input_offset > payload_size || input_length > payload_size - input_offset) { + fillContractViolation(out_error, "functional parser emitted an out-of-bounds object splice"); + return false; + } + if ((canonical_wire.data == nullptr && canonical_wire.size != 0) || + canonical_wire.size > std::numeric_limits::max()) { + fillContractViolation(out_error, "spliced-object callback received invalid canonical wire bytes"); + return false; + } + + try { + const auto type = static_cast(object_type); + auto object = deserializeBuiltinObject(type, canonical_wire.data, static_cast(canonical_wire.size)); + if (!object) { + fillContractViolation(out_error, "spliced canonical object wire is malformed: " + object.error()); + return false; + } + const size_t offset = static_cast(input_offset); + const size_t length = static_cast(input_length); + Span materialized; + sdk::BufferAnchor anchor = state->input_anchor; + if (anchor) { + const uint8_t* data = state->input_payload.data(); + materialized = Span(data == nullptr ? nullptr : data + offset, length); + } else { + auto owned = std::make_shared>(); + if (length != 0) { + owned->assign( + state->input_payload.begin() + static_cast(offset), + state->input_payload.begin() + static_cast(offset + length)); + } + materialized = Span(owned->data(), owned->size()); + anchor = std::move(owned); + } + const auto attach = [&]() { + auto* typed = std::any_cast(&*object); + if (typed == nullptr) { + return false; + } + typed->data = materialized; + typed->anchor = anchor; + return true; + }; + bool attached = false; + switch (type) { + case sdk::BuiltinObjectType::kImage: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kPointCloud: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kDepthImage: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kOccupancyGrid: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kCompressedPointCloud: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kMesh3D: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kVideoFrame: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kOccupancyGridUpdate: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kVoxelGrid: + attached = attach.template operator()(); + break; + default: + break; + } + if (!attached) { + fillContractViolation(out_error, "functional parser splice object type is not materializable"); + return false; + } + state->record = sdk::ObjectRecord{ + .ts = has_timestamp ? std::optional(timestamp_ns) : std::nullopt, + .object = std::move(*object), + }; + return true; + } catch (const std::exception& e) { + sdk::fillError(out_error, 1, "host", std::string("spliced canonical object sink threw: ") + e.what()); + return false; + } catch (...) { + sdk::fillError(out_error, 1, "host", "unknown exception in spliced canonical object sink"); + return false; + } + } + + static void fillContractViolation(PJ_error_t* out_error, std::string_view message) noexcept { + sdk::fillError(out_error, 2, "host", message); + sdk::setExtended(out_error, PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION, nullptr); + } + + static bool validateExpectedObjectType( + const ObjectSinkState& state, uint16_t object_type, PJ_error_t* out_error) noexcept { + if (!state.expected_object_type.has_value() || *state.expected_object_type == sdk::BuiltinObjectType::kNone || + static_cast(*state.expected_object_type) == object_type) { + return true; + } + fillContractViolation( + out_error, "functional parser emitted an object type that differs from the bound classification"); + return false; + } + const PJ_message_parser_vtable_t* vt_ = nullptr; void* ctx_ = nullptr; std::shared_ptr library_owner_; + mutable std::optional expected_object_type_; }; } // namespace PJ diff --git a/pj_plugins/include/pj_plugins/host/native_parser_module.hpp b/pj_plugins/include/pj_plugins/host/native_parser_module.hpp new file mode 100644 index 00000000..2930decf --- /dev/null +++ b/pj_plugins/include/pj_plugins/host/native_parser_module.hpp @@ -0,0 +1,54 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file native_parser_module.hpp + * @brief Session-lifetime loader for native functional parser modules. + * + * Loading resolves the complete frozen module ABI and copies the module's + * embedded manifest. Successfully opened artifacts remain loaded for the + * process session; no module code or manifest pointer is used after unload. + * Manifest admission remains an explicit ParserClaimCatalog caller step. + */ + +#include +#include +#include + +#include "pj_base/diagnostic_sink.hpp" +#include "pj_base/expected.hpp" + +namespace PJ { + +namespace detail { +struct NativeParserModuleState; +} + +class NativeParserModuleInstance; + +class NativeParserModule { + public: + NativeParserModule() = default; + + /// Open and validate one native parser-module artifact. Each rejected load + /// emits exactly one error diagnostic when a sink is supplied. + [[nodiscard]] static Expected load( + std::string_view path, DiagnosticSink sink = {}, std::string diagnostic_source = "NativeParserModule"); + + [[nodiscard]] bool valid() const noexcept { + return state_ != nullptr; + } + + [[nodiscard]] std::string_view path() const noexcept; + [[nodiscard]] std::string_view manifestJson() const noexcept; + + private: + explicit NativeParserModule(std::shared_ptr state); + + std::shared_ptr state_; + + friend class NativeParserModuleInstance; +}; + +} // namespace PJ diff --git a/pj_plugins/include/pj_plugins/host/parser_claim_catalog.hpp b/pj_plugins/include/pj_plugins/host/parser_claim_catalog.hpp new file mode 100644 index 00000000..153cfea3 --- /dev/null +++ b/pj_plugins/include/pj_plugins/host/parser_claim_catalog.hpp @@ -0,0 +1,145 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file parser_claim_catalog.hpp + * @brief Host-owned parser claims, admission, and manifest decoding. + * + * Claims from parser plugins and functional modules share one validated value + * model. Artifact metadata supplies identity and coverage; the host supplies + * provenance and provider generation when it admits a batch. Batch admission + * is transactional so one invalid or duplicate claim rejects the whole batch. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin/builtin_object.hpp" +#include "pj_base/diagnostic_sink.hpp" +#include "pj_base/expected.hpp" +#include "pj_base/parser_route_claims_protocol.h" + +namespace PJ { + +/// Host-derived trust tier. Numeric order is the route-selection order. +enum class ParserClaimProvenance : uint8_t { + kFolderDrop = 0, + kMarketplace = 1, + kBundled = 2, +}; + +/// One admitted parser route claim. +struct ParserClaim { + std::string encoding; + std::string type_name; + uint16_t route_flags = 0; + std::optional object_type; + std::set schema_digests; + std::string provider_id; + std::string claim_id; + int32_t priority = 0; + ParserClaimProvenance provenance = ParserClaimProvenance::kFolderDrop; +}; + +/// Catalog storage for a claim plus the host's immutable provider generation. +struct ParserClaimEntry { + ParserClaim claim; + uint64_t provider_generation = 0; +}; + +/// Parsed module metadata. Claim order remains manifest order. +struct ParserModuleManifest { + std::string id; + std::string name; + std::string version; + std::vector claims; +}; + +/// One successful route-classification result supplied by a parser-plugin host. +struct ParserPluginExactClaim { + std::string encoding; + std::string type_name; + PJ_route_classification_v1_t classification{}; + std::set schema_digests; +}; + +/// The case-sensitive encoding vocabulary accepted by catalog admission. +[[nodiscard]] std::span registeredParserEncodings() noexcept; + +/// Return whether encoding is in the SDK-owned case-sensitive registry. +[[nodiscard]] bool isRegisteredParserEncoding(std::string_view encoding) noexcept; + +/// Normalize one type name according to its registered encoding. +/// +/// ros2msg accepts `pkg/Type` and `pkg/msg/Type`, producing the latter. +/// protobuf accepts an optional leading dot and produces a full dotted name. +/// Other registered encodings preserve a non-empty name verbatim. The wildcard +/// spelling `*` is preserved for every encoding. +[[nodiscard]] Expected normalizeParserTypeName(std::string_view encoding, std::string_view type_name); + +/// Decode and validate one complete parser-module manifest. Provenance is +/// supplied by the host and is never read from JSON. +[[nodiscard]] Expected decodeParserModuleManifest( + std::string_view manifest_json, ParserClaimProvenance provenance); + +/// Synthesize the frozen wildcard and exact parser-plugin claim identities. +/// Declined classifications add no exact claim. Malformed classification +/// records reject the complete synthesized batch. +[[nodiscard]] Expected> synthesizeParserPluginClaims( + std::string_view provider_id, std::span manifest_encodings, + std::span exact_claims, ParserClaimProvenance provenance); + +/// Non-thread-safe host catalog for validated parser claims. +class ParserClaimCatalog { + public: + explicit ParserClaimCatalog(DiagnosticSink sink = {}, std::string diagnostic_source = "ParserClaimCatalog"); + + /// Replace the optional diagnostic sink. + void setDiagnosticSink(DiagnosticSink sink); + + /// Validate and atomically admit a claim batch. The trusted provenance + /// argument overwrites every input value; artifacts cannot choose their tier. + [[nodiscard]] Status admitClaims( + std::vector claims, ParserClaimProvenance provenance, uint64_t provider_generation); + + /// Decode and atomically admit a complete module manifest. + [[nodiscard]] Expected ingestModuleManifest( + std::string_view manifest_json, ParserClaimProvenance provenance, uint64_t provider_generation); + + /// Synthesize and atomically admit all claims for one parser plugin. + [[nodiscard]] Status admitParserPlugin( + std::string_view provider_id, std::span manifest_encodings, + std::span exact_claims, ParserClaimProvenance provenance, + uint64_t provider_generation); + + /// Remove every claim owned by provider_id. Returns true when state changed. + [[nodiscard]] bool removeProvider(std::string_view provider_id); + + /// Remove every claim. Does not advance generation when already empty. + void clear(); + + [[nodiscard]] const std::vector& claims() const noexcept { + return claims_; + } + + /// Monotonic catalog mutation generation, advanced once per non-empty batch. + [[nodiscard]] uint64_t generation() const noexcept { + return generation_; + } + + private: + void report(DiagnosticLevel level, std::string_view id, std::string message) const; + + DiagnosticSink sink_; + std::string diagnostic_source_; + std::vector claims_; + uint64_t generation_ = 0; +}; + +} // namespace PJ diff --git a/pj_plugins/include/pj_plugins/host/parser_module_runtime.hpp b/pj_plugins/include/pj_plugins/host/parser_module_runtime.hpp new file mode 100644 index 00000000..dbccc8e4 --- /dev/null +++ b/pj_plugins/include/pj_plugins/host/parser_module_runtime.hpp @@ -0,0 +1,165 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file parser_module_runtime.hpp + * @brief Native parser-module instance lifecycle and fault classification. + * + * The wrapper performs one serialized create/bind/parse/destroy lifecycle. + * Module-owned descriptor views are decoded and copied before parse returns. + * The independent strike tracker is deliberately pure, non-thread-safe state + * so a host executor can apply its own scheduling and generation policy. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin/builtin_object.hpp" +#include "pj_base/expected.hpp" +#include "pj_base/parser_module_abi.h" +#include "pj_plugins/host/native_parser_module.hpp" + +namespace PJ { + +enum class ParserModuleFaultKind : uint8_t { + kNone, + kDataError, + kContractViolation, +}; + +enum class ParserModuleBindOutcome : uint8_t { + kAccept, + kDecline, + kError, +}; + +struct ParserModuleBindResult { + ParserModuleBindOutcome outcome = ParserModuleBindOutcome::kError; + ParserModuleFaultKind fault = ParserModuleFaultKind::kNone; + int32_t result_code = PJ_MODULE_ERR_GENERIC; + std::string message; +}; + +using ParserModuleScalarValue = std::variant; + +struct ParserModuleScalarField { + std::string name; + ParserModuleScalarValue value; +}; + +struct ParserModuleScalarOutput { + bool has_timestamp = false; + int64_t timestamp_ns = 0; + std::vector fields; +}; + +struct ParserModuleObjectSplice { + uint32_t field_number = 0; + uint64_t input_offset = 0; + std::vector payload_bytes; +}; + +struct ParserModuleObjectOutput { + sdk::BuiltinObject object; + std::vector wire; + std::optional splice; +}; + +using ParserModuleOutput = std::variant; + +struct ParserModuleParseResult { + ParserModuleFaultKind fault = ParserModuleFaultKind::kNone; + int32_t result_code = PJ_MODULE_OK; + std::string message; + std::optional output; +}; + +/// Move-only owner of one native module instance token. +class NativeParserModuleInstance { + public: + NativeParserModuleInstance() = default; + ~NativeParserModuleInstance(); + + NativeParserModuleInstance(NativeParserModuleInstance&& other) noexcept; + NativeParserModuleInstance& operator=(NativeParserModuleInstance&& other) noexcept; + + NativeParserModuleInstance(const NativeParserModuleInstance&) = delete; + NativeParserModuleInstance& operator=(const NativeParserModuleInstance&) = delete; + + /// Create the manifest claim at claim_index. Token-zero creation diagnostics + /// are copied into the returned error. + [[nodiscard]] static Expected create( + const NativeParserModule& module, uint32_t claim_index); + + /// Bind this instance using the frozen BindingInfo v1 codec. + [[nodiscard]] Expected bind(const parser_module::BindingInfoV1& info); + + /// Parse one message and consume the returned descriptor transactionally. + [[nodiscard]] Expected parse(const parser_module::ParseInputV1& input); + + [[nodiscard]] bool valid() const noexcept { + return token_ != PJ_MODULE_CREATION_ERROR_TOKEN; + } + + [[nodiscard]] uint32_t claimIndex() const noexcept { + return claim_index_; + } + + private: + NativeParserModuleInstance( + std::shared_ptr module, uint64_t token, uint32_t claim_index); + + void reset() noexcept; + + // module_ and token_ are always set and cleared together, so a valid() token + // implies a non-null module_. + std::shared_ptr module_; + uint64_t token_ = PJ_MODULE_CREATION_ERROR_TOKEN; + uint32_t claim_index_ = 0; + parser_module::Route bound_route_ = parser_module::Route::kScalar; + uint16_t expected_object_type_ = 0; + bool bound_ = false; +}; + +struct ParserModuleClaimKey { + std::string module_id; + std::string claim_id; + + auto operator<=>(const ParserModuleClaimKey&) const = default; +}; + +enum class ParserModuleClaimHealth : uint8_t { + kActive, + kQuarantined, + kDisabled, +}; + +struct ParserModuleStrikeState { + ParserModuleClaimHealth health = ParserModuleClaimHealth::kActive; + uint8_t strikes = 0; + uint8_t quarantine_count = 0; +}; + +/// Pure per-(module, claim) contract-fault state. Data errors never mutate it. +class ParserModuleStrikeTracker { + public: + [[nodiscard]] ParserModuleStrikeState recordFault(const ParserModuleClaimKey& key, ParserModuleFaultKind fault); + + /// Reactivate a first-time quarantine after the caller successfully replays + /// create and bind for the same immutable binding descriptor. + [[nodiscard]] bool markRecreated(const ParserModuleClaimKey& key); + + [[nodiscard]] ParserModuleStrikeState state(const ParserModuleClaimKey& key) const; + + private: + std::map states_; +}; + +} // namespace PJ diff --git a/pj_plugins/include/pj_plugins/host/parser_route_resolver.hpp b/pj_plugins/include/pj_plugins/host/parser_route_resolver.hpp new file mode 100644 index 00000000..43ef7bc3 --- /dev/null +++ b/pj_plugins/include/pj_plugins/host/parser_route_resolver.hpp @@ -0,0 +1,169 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +/** + * @file parser_route_resolver.hpp + * @brief Deterministic host-side parser route selection and probe caching. + * + * The resolver owns policy and cached probe decisions only. A caller-provided + * callback performs provider-specific creation, binding, and classification on + * the required executor. An optional opaque lease in an accepted decision is + * retained by the cache and returned with the winner. Scalar and object probe + * caches are independent partitions so each route retains its own instance. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/diagnostic_sink.hpp" +#include "pj_base/expected.hpp" +#include "pj_plugins/host/parser_claim_catalog.hpp" + +namespace PJ { + +enum class ParserRoute : uint8_t { + kScalar = 1, + kObject = 2, +}; + +struct ParserRoutePins { + std::optional scalar_provider; + std::optional object_provider; + + [[nodiscard]] const std::optional& forRoute(ParserRoute route) const noexcept { + return route == ParserRoute::kScalar ? scalar_provider : object_provider; + } +}; + +struct ParserRouteRequest { + std::string encoding; + std::string type_name; + std::string schema_digest; + ParserRoute route = ParserRoute::kScalar; +}; + +struct ParserClaimIdentity { + std::string provider_id; + std::string claim_id; + + auto operator<=>(const ParserClaimIdentity&) const = default; +}; + +struct ParserRouteCandidate { + ParserClaim claim; + uint64_t provider_generation = 0; +}; + +enum class ParserProbeOutcome : uint8_t { + kAccept, + kDecline, + kError, +}; + +struct ParserProbeRequest { + const ParserClaim& claim; + uint64_t provider_generation = 0; + std::string_view config_json; + std::string_view config_digest; +}; + +struct ParserProbeDecision { + ParserProbeOutcome outcome = ParserProbeOutcome::kError; + std::string diagnostic; + std::shared_ptr retained_instance; +}; + +using ParserProbeCallback = std::function; + +/// Provider config supplied by the host's config-envelope layer. +struct ParserProviderConfig { + std::string json; + std::string digest; +}; + +using ParserProviderConfigLookup = std::function; + +enum class ParserSelectionTraceKind : uint8_t { + kCandidate, + kCacheHit, + kProbeAccept, + kProbeDecline, + kProbeError, + kAmbiguityTieBreak, + kSelected, + kPinnedProviderUnavailable, + kExhausted, +}; + +struct ParserSelectionTraceEntry { + ParserSelectionTraceKind kind = ParserSelectionTraceKind::kCandidate; + ParserClaimIdentity claim; + std::string detail; +}; + +enum class ParserRouteResolutionStatus : uint8_t { + kSelected, + kNoCandidates, + kNoProviderAccepted, + kPinnedProviderUnavailable, + kPinnedProviderRejected, +}; + +struct ParserRouteResolution { + ParserRouteResolutionStatus status = ParserRouteResolutionStatus::kNoCandidates; + std::optional winner; + std::optional winning_claim; + std::shared_ptr retained_instance; + std::vector ordered_candidates; + std::vector trace; +}; + +/// Deterministic, non-thread-safe route resolver. +class ParserRouteResolver { + public: + explicit ParserRouteResolver(DiagnosticSink sink = {}, std::string diagnostic_source = "ParserRouteResolver"); + ~ParserRouteResolver(); + + ParserRouteResolver(const ParserRouteResolver&) = delete; + ParserRouteResolver& operator=(const ParserRouteResolver&) = delete; + + void setDiagnosticSink(DiagnosticSink sink); + + /// Produce the policy-ordered candidates without probing them. + [[nodiscard]] Expected> orderedCandidates( + const ParserRouteRequest& request, const ParserClaimCatalog& catalog, const ParserRoutePins& pins) const; + + /// Probe candidates in policy order and return a machine-readable trace. + [[nodiscard]] Expected resolve( + const ParserRouteRequest& request, const ParserClaimCatalog& catalog, const ParserRoutePins& pins, + const ParserProviderConfigLookup& provider_config, const ParserProbeCallback& probe); + + /// Explicit invalidation hooks used by the host mutation paths. + void invalidateCatalog(); + void invalidatePins(); + void invalidateProviderConfig(std::string_view provider_id); + + [[nodiscard]] size_t probeCacheSize() const noexcept; + + private: + struct CacheEntry; + + void clearAllCachedState(); + void report(DiagnosticLevel level, std::string_view id, std::string message) const; + + DiagnosticSink sink_; + std::string diagnostic_source_; + std::vector scalar_probe_cache_; + std::vector object_probe_cache_; + std::vector reported_ambiguities_; +}; + +} // namespace PJ diff --git a/pj_plugins/include/pj_plugins/sdk/detail/message_parser_trampolines.hpp b/pj_plugins/include/pj_plugins/sdk/detail/message_parser_trampolines.hpp index 37be6827..6436d284 100644 --- a/pj_plugins/include/pj_plugins/sdk/detail/message_parser_trampolines.hpp +++ b/pj_plugins/include/pj_plugins/sdk/detail/message_parser_trampolines.hpp @@ -124,6 +124,13 @@ inline const void* MessageParserPluginBase::trampoline_get_plugin_extension(void auto* self = static_cast(ctx); try { std::string_view sv = id.data == nullptr ? std::string_view{} : std::string_view(id.data, id.size); + if (sv == PJ_PARSER_ROUTE_CLAIMS_EXTENSION_V1) { + static const PJ_parser_route_claims_v1_t extension{ + .struct_size = sizeof(PJ_parser_route_claims_v1_t), + .classify_routes = trampoline_classify_routes, + }; + return &extension; + } // Once a schema is bound, advertise the functional route only if THIS // schema has a handler. A mixed-model plugin may register handlers for // some schemas and keep legacy parse() for the rest; gating on "any @@ -143,6 +150,14 @@ inline const void* MessageParserPluginBase::trampoline_get_plugin_extension(void }; return &extension; } + if (sv == PJ_PARSER_FUNCTIONAL_EXTENSION_V2 && functional_route_available) { + static const PJ_parser_functional_v2_t extension{ + .struct_size = sizeof(PJ_parser_functional_v2_t), + .parse_scalars = trampoline_parse_scalars_functional_v2, + .parse_object = trampoline_parse_object_functional_v2, + }; + return &extension; + } return self->pluginExtension(sv); } catch (...) { return nullptr; @@ -184,6 +199,54 @@ inline bool MessageParserPluginBase::trampoline_parse_scalars_functional( } } +inline bool MessageParserPluginBase::trampoline_parse_scalars_functional_v2( + void* ctx, int64_t timestamp_ns, PJ_bytes_view_t payload, const PJ_parser_scalar_sink_v1_t* sink, + PJ_error_t* out_error) noexcept { + if (ctx == nullptr) { + storeErrorKind( + out_error, 2, "plugin", "parse_scalars called with null plugin context", + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + auto* self = static_cast(ctx); + if (sink == nullptr || sink->struct_size < PJ_PARSER_SCALAR_SINK_V1_MIN_SIZE || sink->accept_record == nullptr) { + self->storeErrorKind( + out_error, 2, "plugin", "parse_scalars called with invalid scalar sink", + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + if (payload.data == nullptr && payload.size != 0) { + self->storeErrorKind( + out_error, 2, "plugin", "parse_scalars called with invalid payload", PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + + try { + auto record = self->parseScalars(timestamp_ns, Span(payload.data, payload.size)); + if (!record) { + self->storeErrorKind(out_error, 1, "plugin", std::move(record).error(), PJ_PARSER_ERROR_KIND_DATA_ERROR); + return false; + } + const auto fields = sdk::toAbiNamed(Span(record->fields)); + const bool accepted = sink->accept_record( + sink->ctx, record->ts.has_value(), record->ts.value_or(0), fields.data(), fields.size(), out_error); + if (!accepted && (out_error == nullptr || + std::string_view(out_error->extended_kind) != PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION)) { + sdk::setExtended(out_error, PJ_PARSER_ERROR_KIND_SINK_REJECTED, nullptr); + } + return accepted; + } catch (const std::exception& e) { + self->storeErrorKind( + out_error, 1, "plugin", std::string("parse_scalars threw: ") + e.what(), + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } catch (...) { + self->storeErrorKind( + out_error, 1, "plugin", "unknown exception in parse_scalars", PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } +} + inline bool MessageParserPluginBase::trampoline_parse_object_functional( void* ctx, int64_t timestamp_ns, PJ_payload_t payload, const PJ_parser_object_sink_v1_t* sink, PJ_error_t* out_error) noexcept { @@ -246,6 +309,139 @@ inline bool MessageParserPluginBase::trampoline_parse_object_functional( } } +inline bool MessageParserPluginBase::trampoline_parse_object_functional_v2( + void* ctx, int64_t timestamp_ns, PJ_payload_t payload, const PJ_parser_object_sink_v2_t* sink, + PJ_error_t* out_error) noexcept { + if (payload.anchor.ctx != nullptr && payload.anchor.release == nullptr) { + storeErrorKind( + out_error, 2, "plugin", "parse_object payload anchor has context without release callback", + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + + // Functional v2 preserves v1's one-reference ownership rule. Builtin + // handlers materialize complete canonical wire in this revision, so only + // accept_object is invoked; accept_object_spliced remains available to + // module-aware or future handlers. + std::shared_ptr payload_owner; + if (payload.anchor.release != nullptr) { + try { + payload_owner = std::shared_ptr(payload.anchor.ctx, payload.anchor.release); + } catch (const std::exception& e) { + storeErrorKind( + out_error, 1, "plugin", std::string("payload anchor adoption failed: ") + e.what(), + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } catch (...) { + storeErrorKind( + out_error, 1, "plugin", "unknown exception adopting payload anchor", PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + } + if (ctx == nullptr) { + storeErrorKind( + out_error, 2, "plugin", "parse_object called with null plugin context", + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + auto* self = static_cast(ctx); + if (sink == nullptr || sink->struct_size < PJ_PARSER_OBJECT_SINK_V2_MIN_SIZE || sink->accept_object == nullptr || + sink->accept_object_spliced == nullptr) { + self->storeErrorKind( + out_error, 2, "plugin", "parse_object called with invalid v2 object sink", + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + if (payload.data == nullptr && payload.size != 0) { + self->storeErrorKind( + out_error, 2, "plugin", "parse_object called with invalid payload", PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + + try { + sdk::PayloadView payload_view{Span(payload.data, payload.size), std::move(payload_owner)}; + auto record = self->parseObject(timestamp_ns, std::move(payload_view)); + if (!record) { + self->storeErrorKind(out_error, 1, "plugin", std::move(record).error(), PJ_PARSER_ERROR_KIND_DATA_ERROR); + return false; + } + const auto object_type = sdk::typeOf(record->object); + auto canonical_wire = serializeBuiltinObject(record->object); + if (!canonical_wire) { + self->storeErrorKind( + out_error, 1, "plugin", std::move(canonical_wire).error(), PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } + const bool accepted = sink->accept_object( + sink->ctx, record->ts.has_value(), record->ts.value_or(0), static_cast(object_type), + PJ_bytes_view_t{canonical_wire->data(), canonical_wire->size()}, out_error); + if (!accepted && (out_error == nullptr || + std::string_view(out_error->extended_kind) != PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION)) { + sdk::setExtended(out_error, PJ_PARSER_ERROR_KIND_SINK_REJECTED, nullptr); + } + return accepted; + } catch (const std::exception& e) { + self->storeErrorKind( + out_error, 1, "plugin", std::string("parse_object threw: ") + e.what(), + PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } catch (...) { + self->storeErrorKind( + out_error, 1, "plugin", "unknown exception in parse_object", PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + return false; + } +} + +inline bool MessageParserPluginBase::trampoline_classify_routes( + void* ctx, PJ_string_view_t type_name, PJ_bytes_view_t schema, PJ_route_classification_v1_t* out, + PJ_error_t* out_error) noexcept { + if (ctx == nullptr) { + storeError(out_error, 2, "plugin", "classify_routes called with null plugin context"); + return false; + } + auto* self = static_cast(ctx); + if (out == nullptr) { + self->storeError(out_error, 2, "plugin", "classify_routes called with null output"); + return false; + } + if ((type_name.data == nullptr && type_name.size != 0) || (schema.data == nullptr && schema.size != 0)) { + self->storeError(out_error, 2, "plugin", "classify_routes called with an invalid borrowed view"); + return false; + } + + try { + const std::string_view name = + type_name.data == nullptr ? std::string_view{} : std::string_view(type_name.data, type_name.size); + const auto* handler = self->findSchemaHandler(name); + out->match = PJ_PARSER_ROUTE_MATCH_EXACT_V1; + if (handler == nullptr) { + out->route_flags = 0; + out->status = PJ_PARSER_ROUTE_STATUS_DECLINED_V1; + out->object_type = PJ_BUILTIN_OBJECT_TYPE_NONE; + return true; + } + + uint16_t route_flags = 0; + if (handler->parse_scalars) { + route_flags = static_cast(route_flags | PJ_PARSER_ROUTE_FLAG_SCALAR_V1); + } + if (handler->parse_object) { + route_flags = static_cast(route_flags | PJ_PARSER_ROUTE_FLAG_OBJECT_V1); + } + out->route_flags = route_flags; + out->status = PJ_PARSER_ROUTE_STATUS_CLAIMED_V1; + out->object_type = handler->parse_object ? static_cast(handler->object_type) + : static_cast(PJ_BUILTIN_OBJECT_TYPE_NONE); + return true; + } catch (const std::exception& e) { + self->storeError(out_error, 1, "plugin", std::string("classify_routes threw: ") + e.what()); + return false; + } catch (...) { + self->storeError(out_error, 1, "plugin", "unknown exception in classify_routes"); + return false; + } +} + // ----------------------------------------------------------------------------- // Pure-functional API trampolines (builtin-object tail of the vtable) // ----------------------------------------------------------------------------- diff --git a/pj_plugins/include/pj_plugins/sdk/message_parser_plugin_base.hpp b/pj_plugins/include/pj_plugins/sdk/message_parser_plugin_base.hpp index 554f56a0..72ad721b 100644 --- a/pj_plugins/include/pj_plugins/sdk/message_parser_plugin_base.hpp +++ b/pj_plugins/include/pj_plugins/sdk/message_parser_plugin_base.hpp @@ -30,6 +30,7 @@ #include "pj_base/expected.hpp" #include "pj_base/message_parser_protocol.h" #include "pj_base/parser_functional_protocol.h" +#include "pj_base/parser_route_claims_protocol.h" #include "pj_base/plugin_abi_export.hpp" #include "pj_base/sdk/plugin_data_api.hpp" #include "pj_base/sdk/service_registry.hpp" @@ -302,8 +303,9 @@ class MessageParserPluginBase { /// Return a pointer to a static plugin-exposed extension for @p id, or /// nullptr if unknown. Default returns nullptr. The SDK reserves - /// `pj.parser_functional.v1`, which is advertised automatically only after - /// at least one SchemaHandler has been registered. + /// `pj.parser_route_claims.v1` plus `pj.parser_functional.v1` and v2. Route + /// claims are always advertised; both functional revisions are advertised + /// automatically after at least one SchemaHandler has been registered. virtual const void* pluginExtension(std::string_view id) { (void)id; return nullptr; @@ -402,6 +404,12 @@ class MessageParserPluginBase { sdk::fillError(out_error, code, domain, message); } + static void storeErrorKind( + PJ_error_t* out_error, int32_t code, std::string_view domain, std::string_view message, std::string_view kind) { + sdk::fillError(out_error, code, domain, message); + sdk::setExtended(out_error, kind, nullptr); + } + static void trampoline_destroy(void* ctx) noexcept; static bool trampoline_bind(void* ctx, PJ_service_registry_t registry, PJ_error_t* out_error) noexcept; static bool trampoline_bind_schema( @@ -415,9 +423,18 @@ class MessageParserPluginBase { PJ_MESSAGE_PARSER_DSO_LOCAL static bool trampoline_parse_scalars_functional( void* ctx, int64_t timestamp_ns, PJ_bytes_view_t payload, const PJ_parser_scalar_sink_v1_t* sink, PJ_error_t* out_error) noexcept; + PJ_MESSAGE_PARSER_DSO_LOCAL static bool trampoline_parse_scalars_functional_v2( + void* ctx, int64_t timestamp_ns, PJ_bytes_view_t payload, const PJ_parser_scalar_sink_v1_t* sink, + PJ_error_t* out_error) noexcept; PJ_MESSAGE_PARSER_DSO_LOCAL static bool trampoline_parse_object_functional( void* ctx, int64_t timestamp_ns, PJ_payload_t payload, const PJ_parser_object_sink_v1_t* sink, PJ_error_t* out_error) noexcept; + PJ_MESSAGE_PARSER_DSO_LOCAL static bool trampoline_parse_object_functional_v2( + void* ctx, int64_t timestamp_ns, PJ_payload_t payload, const PJ_parser_object_sink_v2_t* sink, + PJ_error_t* out_error) noexcept; + PJ_MESSAGE_PARSER_DSO_LOCAL static bool trampoline_classify_routes( + void* ctx, PJ_string_view_t type_name, PJ_bytes_view_t schema, PJ_route_classification_v1_t* out, + PJ_error_t* out_error) noexcept; static bool trampoline_classify_schema( void* ctx, PJ_string_view_t type_name, PJ_bytes_view_t schema, PJ_schema_classification_t* out_classification, PJ_error_t* out_error) noexcept; diff --git a/pj_plugins/src/detail/native_parser_module_loader.hpp b/pj_plugins/src/detail/native_parser_module_loader.hpp new file mode 100644 index 00000000..8dd4254e --- /dev/null +++ b/pj_plugins/src/detail/native_parser_module_loader.hpp @@ -0,0 +1,81 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include +#include +#include +#include +#include + +#if defined(_WIN32) +#ifndef NOMINMAX +#define NOMINMAX +#endif +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#else +#include +#endif + +#include "pj_base/expected.hpp" + +namespace PJ::detail { + +using NativeModuleHandle = void*; + +inline Expected openNativeParserModule(std::string_view path) { +#if defined(_WIN32) + if (path.size() > static_cast(INT_MAX)) { + return unexpected("native parser-module path is too long"); + } + const int required = + MultiByteToWideChar(CP_UTF8, MB_ERR_INVALID_CHARS, path.data(), static_cast(path.size()), nullptr, 0); + if (required <= 0) { + return unexpected("native parser-module path is not valid UTF-8"); + } + std::wstring wide_path(static_cast(required), L'\0'); + if (MultiByteToWideChar( + CP_UTF8, MB_ERR_INVALID_CHARS, path.data(), static_cast(path.size()), wide_path.data(), required) == 0) { + return unexpected("native parser-module path conversion failed"); + } + HMODULE module = + LoadLibraryExW(wide_path.c_str(), nullptr, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); + if (module == nullptr) { + return unexpected("LoadLibraryExW failed (error " + std::to_string(GetLastError()) + ")"); + } + return reinterpret_cast(module); +#else + void* handle = dlopen(std::string(path).c_str(), RTLD_LOCAL | RTLD_NOW); + if (handle == nullptr) { + const char* error = dlerror(); + return unexpected(error == nullptr ? "dlopen failed" : error); + } + return handle; +#endif +} + +inline Expected resolveNativeParserModuleSymbol(NativeModuleHandle handle, const char* name) { + if (handle == nullptr) { + return unexpected("native parser module is not loaded"); + } +#if defined(_WIN32) + FARPROC symbol = GetProcAddress(reinterpret_cast(handle), name); + if (symbol == nullptr) { + return unexpected(std::string(name) + " not found"); + } + return reinterpret_cast(symbol); +#else + dlerror(); + void* symbol = dlsym(handle, name); + const char* error = dlerror(); + if (error != nullptr) { + return unexpected(std::string(name) + " not found: " + error); + } + return symbol; +#endif +} + +} // namespace PJ::detail diff --git a/pj_plugins/src/detail/native_parser_module_state.hpp b/pj_plugins/src/detail/native_parser_module_state.hpp new file mode 100644 index 00000000..91a7a7e7 --- /dev/null +++ b/pj_plugins/src/detail/native_parser_module_state.hpp @@ -0,0 +1,29 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +#include "detail/native_parser_module_loader.hpp" +#include "pj_base/parser_module_abi.h" + +namespace PJ::detail { + +struct NativeParserModuleState { + NativeModuleHandle handle = nullptr; + std::string path; + std::string manifest_json; + + PJ_module_abi_fn_t abi = nullptr; + PJ_module_create_fn_t create = nullptr; + PJ_module_destroy_fn_t destroy = nullptr; + PJ_module_bind_fn_t bind = nullptr; + PJ_module_parse_fn_t parse = nullptr; + PJ_module_last_error_fn_t last_error = nullptr; + PJ_module_alloc_fn_t alloc = nullptr; + PJ_module_free_fn_t free = nullptr; + PJ_module_manifest_addr_fn_t manifest_addr = nullptr; + PJ_module_manifest_len_fn_t manifest_len = nullptr; +}; + +} // namespace PJ::detail diff --git a/pj_plugins/src/native_parser_module.cpp b/pj_plugins/src/native_parser_module.cpp new file mode 100644 index 00000000..d1830562 --- /dev/null +++ b/pj_plugins/src/native_parser_module.cpp @@ -0,0 +1,124 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/native_parser_module.hpp" + +#include +#include +#include +#include +#include +#include + +#include "detail/native_parser_module_state.hpp" +#include "pj_base/parser_module_abi.h" + +namespace PJ { +namespace { + +std::vector& sessionHandles() { + static auto* handles = new std::vector(); + return *handles; +} + +std::mutex& sessionHandlesMutex() { + static auto* mutex = new std::mutex(); + return *mutex; +} + +void retainForSession(detail::NativeModuleHandle handle) { + std::lock_guard lock(sessionHandlesMutex()); + sessionHandles().push_back(handle); +} + +Expected rejectLoad( + std::string_view path, const DiagnosticSink& sink, std::string_view source, std::string message) { + if (sink) { + sink( + Diagnostic{ + .level = DiagnosticLevel::kError, + .source = std::string(source), + .id = std::string(path), + .message = message, + }); + } + return unexpected(std::move(message)); +} + +template +Expected resolve(detail::NativeModuleHandle handle, const char* name) { + auto symbol = detail::resolveNativeParserModuleSymbol(handle, name); + if (!symbol) { + return unexpected(symbol.error()); + } + return reinterpret_cast(*symbol); +} + +} // namespace + +NativeParserModule::NativeParserModule(std::shared_ptr state) + : state_(std::move(state)) {} + +Expected NativeParserModule::load( + std::string_view path, DiagnosticSink sink, std::string diagnostic_source) { + auto handle_result = detail::openNativeParserModule(path); + if (!handle_result) { + return rejectLoad(path, sink, diagnostic_source, "failed to open native parser module: " + handle_result.error()); + } + const auto handle = *handle_result; + retainForSession(handle); + + auto state = std::make_shared(); + state->handle = handle; + state->path = path; + +#define PJ_RESOLVE_MODULE_EXPORT(member, type, name) \ + do { \ + auto resolved = resolve(handle, name); \ + if (!resolved) { \ + return rejectLoad(path, sink, diagnostic_source, resolved.error()); \ + } \ + state->member = *resolved; \ + } while (false) + + PJ_RESOLVE_MODULE_EXPORT(abi, PJ_module_abi_fn_t, PJ_MODULE_ABI_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(create, PJ_module_create_fn_t, PJ_MODULE_CREATE_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(destroy, PJ_module_destroy_fn_t, PJ_MODULE_DESTROY_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(bind, PJ_module_bind_fn_t, PJ_MODULE_BIND_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(parse, PJ_module_parse_fn_t, PJ_MODULE_PARSE_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(last_error, PJ_module_last_error_fn_t, PJ_MODULE_LAST_ERROR_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(alloc, PJ_module_alloc_fn_t, PJ_MODULE_ALLOC_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(free, PJ_module_free_fn_t, PJ_MODULE_FREE_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(manifest_addr, PJ_module_manifest_addr_fn_t, PJ_MODULE_MANIFEST_ADDR_EXPORT_NAME); + PJ_RESOLVE_MODULE_EXPORT(manifest_len, PJ_module_manifest_len_fn_t, PJ_MODULE_MANIFEST_LEN_EXPORT_NAME); + +#undef PJ_RESOLVE_MODULE_EXPORT + + const uint32_t actual_abi = state->abi(); + if (actual_abi != PJ_PARSER_MODULE_ABI_VERSION) { + return rejectLoad( + path, sink, diagnostic_source, + "native parser module ABI mismatch (expected " + std::to_string(PJ_PARSER_MODULE_ABI_VERSION) + ", got " + + std::to_string(actual_abi) + ")"); + } + + const uint64_t manifest_addr = state->manifest_addr(); + const uint64_t manifest_len = state->manifest_len(); + if (manifest_addr == 0 || manifest_len == 0 || manifest_len > std::numeric_limits::max()) { + return rejectLoad(path, sink, diagnostic_source, "native parser module manifest is unreadable"); + } + const auto* manifest = reinterpret_cast(static_cast(manifest_addr)); + state->manifest_json.assign(manifest, static_cast(manifest_len)); + + return NativeParserModule(std::move(state)); +} + +std::string_view NativeParserModule::path() const noexcept { + return state_ == nullptr ? std::string_view{} : std::string_view(state_->path); +} + +std::string_view NativeParserModule::manifestJson() const noexcept { + return state_ == nullptr ? std::string_view{} : std::string_view(state_->manifest_json); +} + +} // namespace PJ diff --git a/pj_plugins/src/parser_claim_catalog.cpp b/pj_plugins/src/parser_claim_catalog.cpp new file mode 100644 index 00000000..ca7e58d8 --- /dev/null +++ b/pj_plugins/src/parser_claim_catalog.cpp @@ -0,0 +1,564 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_claim_catalog.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module_abi.h" +#include "pj_base/sdk/semver.hpp" + +namespace PJ { +namespace { + +constexpr std::array kRegisteredEncodings = { + "bson", "cbor", "data_tamer", "json", "msgpack", "omgidl", "protobuf", "ros1", "ros1msg", "ros2", "ros2msg", +}; + +[[nodiscard]] bool isIdentifierStart(char value) noexcept { + return (value >= 'A' && value <= 'Z') || (value >= 'a' && value <= 'z') || value == '_'; +} + +[[nodiscard]] bool isIdentifierContinuation(char value) noexcept { + return isIdentifierStart(value) || (value >= '0' && value <= '9'); +} + +[[nodiscard]] bool isProtobufFullName(std::string_view name) noexcept { + size_t component_start = 0; + while (component_start < name.size()) { + const size_t component_end = name.find('.', component_start); + const size_t end = component_end == std::string_view::npos ? name.size() : component_end; + const std::string_view component = name.substr(component_start, end - component_start); + if (component.empty() || !isIdentifierStart(component.front())) { + return false; + } + if (!std::all_of(component.begin() + 1, component.end(), isIdentifierContinuation)) { + return false; + } + if (component_end == std::string_view::npos) { + return true; + } + component_start = component_end + 1; + } + return false; +} + +[[nodiscard]] bool isKnownObjectType(sdk::BuiltinObjectType object_type) noexcept { + if (object_type == sdk::BuiltinObjectType::kNone) { + return false; + } + const auto parsed = sdk::parseBuiltinObjectType(sdk::name(object_type)); + return parsed.has_value() && *parsed == object_type; +} + +[[nodiscard]] bool isSchemaDigest(std::string_view digest) noexcept { + constexpr std::string_view kPrefix = "sha256:"; + if (!digest.starts_with(kPrefix) || digest.size() != kPrefix.size() + 64) { + return false; + } + return std::all_of(digest.begin() + static_cast(kPrefix.size()), digest.end(), [](char value) { + return (value >= '0' && value <= '9') || (value >= 'a' && value <= 'f') || (value >= 'A' && value <= 'F'); + }); +} + +[[nodiscard]] Expected normalizeAndValidateClaim(ParserClaim claim) { + if (claim.provider_id.empty()) { + return unexpected("parser claim provider_id must not be empty"); + } + if (claim.claim_id.empty()) { + return unexpected("parser claim claim_id must not be empty"); + } + if (claim.priority < -1000 || claim.priority > 1000) { + return unexpected( + "parser claim priority is outside the admitted range [-1000,1000]: " + claim.provider_id + "/" + + claim.claim_id); + } + if (!isRegisteredParserEncoding(claim.encoding)) { + return unexpected("parser claim uses unknown encoding: " + claim.encoding); + } + + auto normalized_type = normalizeParserTypeName(claim.encoding, claim.type_name); + if (!normalized_type) { + return unexpected(normalized_type.error()); + } + claim.type_name = std::move(*normalized_type); + + constexpr uint16_t kKnownRouteFlags = PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1; + if (claim.route_flags == 0 || (claim.route_flags & ~kKnownRouteFlags) != 0) { + return unexpected("parser claim route_flags must contain only scalar and/or object"); + } + + const bool has_object_route = (claim.route_flags & PJ_PARSER_ROUTE_FLAG_OBJECT_V1) != 0; + if (claim.type_name == "*" && has_object_route) { + return unexpected("parser claim wildcard type_name cannot claim the object route"); + } + if (has_object_route != claim.object_type.has_value()) { + return unexpected( + has_object_route ? "parser claim object route requires object_type" + : "parser claim object_type is forbidden without the object route"); + } + if (claim.object_type.has_value() && !isKnownObjectType(*claim.object_type)) { + return unexpected("parser claim uses unknown object_type"); + } + for (const auto& digest : claim.schema_digests) { + if (!isSchemaDigest(digest)) { + return unexpected("parser claim schema_digest must use sha256:<64-hex>"); + } + } + return claim; +} + +[[nodiscard]] Expected requiredString( + const nlohmann::json& object, std::string_view key, std::string_view context) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_string() || it->get_ref().empty()) { + return unexpected(std::string(context) + " missing required non-empty string key: " + std::string(key)); + } + return it->get(); +} + +[[nodiscard]] Expected requiredPriority(const nlohmann::json& object, size_t claim_index) { + const auto it = object.find("priority"); + if (it == object.end() || !it->is_number_integer()) { + return unexpected("parser module manifest claim " + std::to_string(claim_index) + " missing integer priority"); + } + + if (it->is_number_unsigned()) { + const uint64_t value = it->get(); + if (value > static_cast(std::numeric_limits::max())) { + return unexpected("parser module manifest claim priority does not fit int32"); + } + return static_cast(value); + } + const int64_t value = it->get(); + if (value < std::numeric_limits::min() || value > std::numeric_limits::max()) { + return unexpected("parser module manifest claim priority does not fit int32"); + } + return static_cast(value); +} + +[[nodiscard]] Expected readRouteFlags(const nlohmann::json& claim, size_t claim_index) { + const auto routes = claim.find("routes"); + if (routes == claim.end() || !routes->is_array() || routes->empty()) { + return unexpected( + "parser module manifest claim " + std::to_string(claim_index) + " requires a non-empty routes array"); + } + + uint16_t flags = 0; + for (const auto& route : *routes) { + if (!route.is_string()) { + return unexpected("parser module manifest claim " + std::to_string(claim_index) + " routes must contain strings"); + } + const auto& name = route.get_ref(); + if (name == "scalar") { + flags |= PJ_PARSER_ROUTE_FLAG_SCALAR_V1; + } else if (name == "object") { + flags |= PJ_PARSER_ROUTE_FLAG_OBJECT_V1; + } else { + return unexpected("parser module manifest claim uses unknown route: " + name); + } + } + return flags; +} + +[[nodiscard]] Expected> readSchemaDigests(const nlohmann::json& claim, size_t claim_index) { + std::set digests; + const auto values = claim.find("schema_digests"); + if (values == claim.end()) { + return digests; + } + if (!values->is_array()) { + return unexpected( + "parser module manifest claim " + std::to_string(claim_index) + " schema_digests must be an array"); + } + for (const auto& value : *values) { + if (!value.is_string()) { + return unexpected( + "parser module manifest claim " + std::to_string(claim_index) + " schema_digests must contain strings"); + } + digests.insert(value.get()); + } + return digests; +} + +[[nodiscard]] std::string claimIdentity(const ParserClaim& claim) { + return claim.provider_id + "/" + claim.claim_id; +} + +} // namespace + +std::span registeredParserEncodings() noexcept { + return kRegisteredEncodings; +} + +bool isRegisteredParserEncoding(std::string_view encoding) noexcept { + return std::find(kRegisteredEncodings.begin(), kRegisteredEncodings.end(), encoding) != kRegisteredEncodings.end(); +} + +Expected normalizeParserTypeName(std::string_view encoding, std::string_view type_name) { + if (!isRegisteredParserEncoding(encoding)) { + return unexpected("cannot normalize type name for unknown encoding: " + std::string(encoding)); + } + if (type_name.empty()) { + return unexpected("parser claim type_name must not be empty"); + } + if (type_name == "*") { + return std::string(type_name); + } + + if (encoding == "ros2msg") { + const size_t first = type_name.find('/'); + if (first == std::string_view::npos || first == 0 || first + 1 == type_name.size()) { + return unexpected("ros2msg type_name must be pkg/Type or pkg/msg/Type"); + } + const size_t second = type_name.find('/', first + 1); + if (second == std::string_view::npos) { + return std::string(type_name.substr(0, first)) + "/msg/" + std::string(type_name.substr(first + 1)); + } + if (type_name.substr(first + 1, second - first - 1) != "msg" || second + 1 == type_name.size() || + type_name.find('/', second + 1) != std::string_view::npos) { + return unexpected("ros2msg type_name must be pkg/Type or pkg/msg/Type"); + } + return std::string(type_name); + } + + if (encoding == "protobuf") { + if (type_name.front() == '.') { + type_name.remove_prefix(1); + } + if (!isProtobufFullName(type_name)) { + return unexpected("protobuf type_name must be a full dotted message name"); + } + return std::string(type_name); + } + + return std::string(type_name); +} + +Expected decodeParserModuleManifest( + std::string_view manifest_json, ParserClaimProvenance provenance) { + if (manifest_json.empty()) { + return unexpected("parser module manifest is empty"); + } + + nlohmann::json json; + try { + json = nlohmann::json::parse(manifest_json); + } catch (const nlohmann::json::exception& error) { + return unexpected("parser module manifest is invalid JSON: " + std::string(error.what())); + } + if (!json.is_object()) { + return unexpected("parser module manifest must be a JSON object"); + } + if (json.contains("provenance")) { + return unexpected("parser module manifest must not declare host-owned provenance"); + } + + const auto abi = json.find("module_abi"); + if (abi == json.end() || !abi->is_number_integer()) { + return unexpected("parser module manifest missing integer module_abi"); + } + // The host ABI version is positive, so any signed (hence negative) encoding + // of module_abi is a mismatch by construction. + if (!abi->is_number_unsigned() || abi->get() != PJ_PARSER_MODULE_ABI_VERSION) { + return unexpected( + "parser module manifest module_abi does not match host ABI " + std::to_string(PJ_PARSER_MODULE_ABI_VERSION)); + } + + auto id = requiredString(json, "id", "parser module manifest"); + if (!id) { + return unexpected(id.error()); + } + auto name = requiredString(json, "name", "parser module manifest"); + if (!name) { + return unexpected(name.error()); + } + auto version = requiredString(json, "version", "parser module manifest"); + if (!version) { + return unexpected(version.error()); + } + if (auto parsed = SemVer::parse(*version); !parsed) { + return unexpected("parser module manifest version is not valid SemVer: " + parsed.error()); + } + + const auto claims_json = json.find("claims"); + if (claims_json == json.end() || !claims_json->is_array()) { + return unexpected("parser module manifest requires a claims array"); + } + + ParserModuleManifest manifest{.id = *id, .name = *name, .version = *version, .claims = {}}; + manifest.claims.reserve(claims_json->size()); + std::set> identities; + + for (size_t index = 0; index < claims_json->size(); ++index) { + const auto& claim_json = (*claims_json)[index]; + if (!claim_json.is_object()) { + return unexpected("parser module manifest claim " + std::to_string(index) + " must be an object"); + } + if (claim_json.contains("provenance")) { + return unexpected("parser module manifest claim must not declare host-owned provenance"); + } + + auto claim_id = requiredString(claim_json, "claim_id", "parser module manifest claim"); + if (!claim_id) { + return unexpected(claim_id.error()); + } + auto encoding = requiredString(claim_json, "encoding", "parser module manifest claim"); + if (!encoding) { + return unexpected(encoding.error()); + } + auto type_name = requiredString(claim_json, "type_name", "parser module manifest claim"); + if (!type_name) { + return unexpected(type_name.error()); + } + auto route_flags = readRouteFlags(claim_json, index); + if (!route_flags) { + return unexpected(route_flags.error()); + } + auto priority = requiredPriority(claim_json, index); + if (!priority) { + return unexpected(priority.error()); + } + auto schema_digests = readSchemaDigests(claim_json, index); + if (!schema_digests) { + return unexpected(schema_digests.error()); + } + + std::optional object_type; + const auto object_type_json = claim_json.find("object_type"); + if (object_type_json != claim_json.end()) { + if (!object_type_json->is_string()) { + return unexpected("parser module manifest claim object_type must be a string"); + } + object_type = sdk::parseBuiltinObjectType(object_type_json->get_ref()); + if (!object_type.has_value() || *object_type == sdk::BuiltinObjectType::kNone) { + return unexpected( + "parser module manifest claim uses unknown object_type: " + object_type_json->get()); + } + } + + ParserClaim claim{ + .encoding = *encoding, + .type_name = *type_name, + .route_flags = *route_flags, + .object_type = object_type, + .schema_digests = std::move(*schema_digests), + .provider_id = manifest.id, + .claim_id = *claim_id, + .priority = *priority, + .provenance = provenance, + }; + auto validated = normalizeAndValidateClaim(std::move(claim)); + if (!validated) { + return unexpected(validated.error()); + } + if (!identities.emplace(validated->provider_id, validated->claim_id).second) { + return unexpected("duplicate parser claim identity: " + claimIdentity(*validated)); + } + manifest.claims.push_back(std::move(*validated)); + } + + return manifest; +} + +Expected> synthesizeParserPluginClaims( + std::string_view provider_id, std::span manifest_encodings, + std::span exact_claims, ParserClaimProvenance provenance) { + if (provider_id.empty()) { + return unexpected("parser plugin provider id must not be empty"); + } + + std::vector claims; + claims.reserve(manifest_encodings.size() + exact_claims.size()); + std::set encodings; + std::set> identities; + + for (const auto& encoding : manifest_encodings) { + ParserClaim wildcard{ + .encoding = encoding, + .type_name = "*", + .route_flags = PJ_PARSER_ROUTE_FLAG_SCALAR_V1, + .object_type = std::nullopt, + .schema_digests = {}, + .provider_id = std::string(provider_id), + .claim_id = "wildcard:" + encoding, + .priority = 0, + .provenance = provenance, + }; + auto validated = normalizeAndValidateClaim(std::move(wildcard)); + if (!validated) { + return unexpected(validated.error()); + } + if (!encodings.insert(encoding).second || !identities.emplace(validated->provider_id, validated->claim_id).second) { + return unexpected("duplicate parser claim identity: " + claimIdentity(*validated)); + } + claims.push_back(std::move(*validated)); + } + + for (const auto& exact : exact_claims) { + if (!encodings.contains(exact.encoding)) { + return unexpected("parser exact claim encoding is absent from its manifest: " + exact.encoding); + } + if (exact.classification.match != PJ_PARSER_ROUTE_MATCH_EXACT_V1) { + return unexpected("parser route classification reported a non-exact match"); + } + if (exact.classification.status == PJ_PARSER_ROUTE_STATUS_DECLINED_V1) { + if (exact.classification.route_flags != 0 || + exact.classification.object_type != static_cast(sdk::BuiltinObjectType::kNone)) { + return unexpected("declined parser route classification contains claimed route data"); + } + continue; + } + if (exact.classification.status != PJ_PARSER_ROUTE_STATUS_CLAIMED_V1) { + return unexpected("parser route classification has an invalid status"); + } + if (exact.type_name == "*") { + return unexpected("parser route classification extension cannot report wildcard claims"); + } + + std::optional object_type; + if ((exact.classification.route_flags & PJ_PARSER_ROUTE_FLAG_OBJECT_V1) != 0) { + object_type = static_cast(exact.classification.object_type); + } else if (exact.classification.object_type != static_cast(sdk::BuiltinObjectType::kNone)) { + return unexpected("parser route classification has object_type without an object route"); + } + + auto normalized_type = normalizeParserTypeName(exact.encoding, exact.type_name); + if (!normalized_type) { + return unexpected(normalized_type.error()); + } + ParserClaim claim{ + .encoding = exact.encoding, + .type_name = *normalized_type, + .route_flags = exact.classification.route_flags, + .object_type = object_type, + .schema_digests = exact.schema_digests, + .provider_id = std::string(provider_id), + .claim_id = "handler:" + exact.encoding + ":" + *normalized_type, + .provenance = provenance, + }; + auto validated = normalizeAndValidateClaim(std::move(claim)); + if (!validated) { + return unexpected(validated.error()); + } + if (!identities.emplace(validated->provider_id, validated->claim_id).second) { + return unexpected("duplicate parser claim identity: " + claimIdentity(*validated)); + } + claims.push_back(std::move(*validated)); + } + + return claims; +} + +ParserClaimCatalog::ParserClaimCatalog(DiagnosticSink sink, std::string diagnostic_source) + : sink_(std::move(sink)), diagnostic_source_(std::move(diagnostic_source)) {} + +void ParserClaimCatalog::setDiagnosticSink(DiagnosticSink sink) { + sink_ = std::move(sink); +} + +Status ParserClaimCatalog::admitClaims( + std::vector claims, ParserClaimProvenance provenance, uint64_t provider_generation) { + std::vector validated_claims; + validated_claims.reserve(claims.size()); + std::set> batch_identities; + + for (auto& claim : claims) { + claim.provenance = provenance; + auto validated = normalizeAndValidateClaim(std::move(claim)); + if (!validated) { + report(DiagnosticLevel::kError, {}, validated.error()); + return unexpected(validated.error()); + } + const auto identity = std::pair(validated->provider_id, validated->claim_id); + if (!batch_identities.insert(identity).second) { + const std::string error = "duplicate parser claim identity: " + claimIdentity(*validated); + report(DiagnosticLevel::kError, validated->provider_id, error); + return unexpected(error); + } + const bool already_present = std::any_of(claims_.begin(), claims_.end(), [&](const ParserClaimEntry& entry) { + return entry.claim.provider_id == identity.first && entry.claim.claim_id == identity.second; + }); + if (already_present) { + const std::string error = "duplicate parser claim identity: " + claimIdentity(*validated); + report(DiagnosticLevel::kError, validated->provider_id, error); + return unexpected(error); + } + validated_claims.push_back(std::move(*validated)); + } + + for (auto& claim : validated_claims) { + claims_.push_back(ParserClaimEntry{.claim = std::move(claim), .provider_generation = provider_generation}); + } + if (!validated_claims.empty()) { + ++generation_; + } + return okStatus(); +} + +Expected ParserClaimCatalog::ingestModuleManifest( + std::string_view manifest_json, ParserClaimProvenance provenance, uint64_t provider_generation) { + auto manifest = decodeParserModuleManifest(manifest_json, provenance); + if (!manifest) { + report(DiagnosticLevel::kError, {}, manifest.error()); + return unexpected(manifest.error()); + } + auto admission = admitClaims(manifest->claims, provenance, provider_generation); + if (!admission) { + return unexpected(admission.error()); + } + return manifest; +} + +Status ParserClaimCatalog::admitParserPlugin( + std::string_view provider_id, std::span manifest_encodings, + std::span exact_claims, ParserClaimProvenance provenance, + uint64_t provider_generation) { + auto claims = synthesizeParserPluginClaims(provider_id, manifest_encodings, exact_claims, provenance); + if (!claims) { + report(DiagnosticLevel::kError, provider_id, claims.error()); + return unexpected(claims.error()); + } + return admitClaims(std::move(*claims), provenance, provider_generation); +} + +bool ParserClaimCatalog::removeProvider(std::string_view provider_id) { + const size_t prior_size = claims_.size(); + std::erase_if(claims_, [&](const ParserClaimEntry& entry) { return entry.claim.provider_id == provider_id; }); + if (claims_.size() == prior_size) { + return false; + } + ++generation_; + return true; +} + +void ParserClaimCatalog::clear() { + if (claims_.empty()) { + return; + } + claims_.clear(); + ++generation_; +} + +void ParserClaimCatalog::report(DiagnosticLevel level, std::string_view id, std::string message) const { + if (!sink_) { + return; + } + sink_( + Diagnostic{ + .level = level, + .source = diagnostic_source_, + .id = std::string(id), + .message = std::move(message), + .timestamp = std::chrono::system_clock::now(), + }); +} + +} // namespace PJ diff --git a/pj_plugins/src/parser_module_runtime.cpp b/pj_plugins/src/parser_module_runtime.cpp new file mode 100644 index 00000000..79dc3fda --- /dev/null +++ b/pj_plugins/src/parser_module_runtime.cpp @@ -0,0 +1,375 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_module_runtime.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "detail/native_parser_module_state.hpp" +#include "pj_base/builtin/builtin_object_codec.hpp" +#include "pj_base/builtin_object_abi.h" +#include "pj_base/span.hpp" + +namespace PJ { +namespace { + +uint64_t addressOf(const void* pointer) { + return static_cast(reinterpret_cast(pointer)); +} + +Expected copyLastError(const detail::NativeParserModuleState& module, uint64_t token) { + std::array buffer{}; + const uint64_t written = module.last_error(token, addressOf(buffer.data()), buffer.size()); + if (written > buffer.size()) { + return unexpected("pj_module_last_error returned a length larger than its buffer"); + } + const auto length = static_cast(written); + const auto terminator = std::find(buffer.begin(), buffer.begin() + static_cast(length), '\0'); + return std::string(buffer.begin(), terminator); +} + +ParserModuleParseResult contractViolation(int32_t code, std::string message) { + return ParserModuleParseResult{ + .fault = ParserModuleFaultKind::kContractViolation, + .result_code = code, + .message = std::move(message), + .output = std::nullopt, + }; +} + +Expected ownScalarOutput(const parser_module::ScalarOutputV1& scalar) { + ParserModuleScalarOutput owned; + owned.has_timestamp = scalar.has_timestamp; + owned.timestamp_ns = scalar.timestamp_ns; + owned.fields.reserve(scalar.fields.size()); + for (const auto& field : scalar.fields) { + ParserModuleScalarValue value = std::visit( + [](const Value& item) -> ParserModuleScalarValue { + if constexpr (std::is_same_v) { + return std::string(item); + } else { + return item; + } + }, + field.value); + owned.fields.push_back(ParserModuleScalarField{.name = std::string(field.name), .value = std::move(value)}); + } + return owned; +} + +Expected ownObjectOutput( + const parser_module::ObjectOutputV1& object, Span input_payload, uint16_t expected_type) { + if (object.object_type != expected_type) { + return unexpected( + "output object type " + std::to_string(object.object_type) + " does not match bound type " + + std::to_string(expected_type)); + } + + const auto type = static_cast(object.object_type); + auto decoded = deserializeBuiltinObject(type, object.wire.data(), object.wire.size()); + if (!decoded) { + return unexpected("output canonical wire is malformed: " + decoded.error()); + } + + ParserModuleObjectOutput owned; + owned.object = std::move(*decoded); + owned.wire.assign(object.wire.begin(), object.wire.end()); + if (object.splice.has_value()) { + uint32_t eligible_field = 0; + if (!pj_builtin_object_splice_field_number_v1(object.object_type, &eligible_field) || + eligible_field != object.splice->field_number) { + return unexpected("output splice field is not eligible for the object type"); + } + const uint64_t payload_size = static_cast(input_payload.size()); + if (object.splice->input_offset > payload_size || + object.splice->input_length > payload_size - object.splice->input_offset) { + return unexpected("output splice range is outside the parse payload"); + } + const auto offset = static_cast(object.splice->input_offset); + const auto length = static_cast(object.splice->input_length); + auto materialized = std::make_shared>( + input_payload.begin() + static_cast(offset), + input_payload.begin() + static_cast(offset + length)); + const auto attach = [&]() -> bool { + auto* typed = std::any_cast(&owned.object); + if (typed == nullptr) { + return false; + } + typed->data = Span(materialized->data(), materialized->size()); + typed->anchor = materialized; + return true; + }; + bool attached = false; + switch (type) { + case sdk::BuiltinObjectType::kImage: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kPointCloud: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kDepthImage: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kOccupancyGrid: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kCompressedPointCloud: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kMesh3D: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kVideoFrame: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kOccupancyGridUpdate: + attached = attach.template operator()(); + break; + case sdk::BuiltinObjectType::kVoxelGrid: + attached = attach.template operator()(); + break; + default: + break; + } + if (!attached) { + return unexpected("output splice could not be attached to its canonical object"); + } + owned.splice = ParserModuleObjectSplice{ + .field_number = object.splice->field_number, + .input_offset = object.splice->input_offset, + .payload_bytes = *materialized, + }; + } + return owned; +} + +} // namespace + +NativeParserModuleInstance::NativeParserModuleInstance( + std::shared_ptr module, uint64_t token, uint32_t claim_index) + : module_(std::move(module)), token_(token), claim_index_(claim_index) {} + +NativeParserModuleInstance::~NativeParserModuleInstance() { + reset(); +} + +NativeParserModuleInstance::NativeParserModuleInstance(NativeParserModuleInstance&& other) noexcept + : module_(std::move(other.module_)), + token_(std::exchange(other.token_, PJ_MODULE_CREATION_ERROR_TOKEN)), + claim_index_(other.claim_index_), + bound_route_(other.bound_route_), + expected_object_type_(other.expected_object_type_), + bound_(other.bound_) { + other.bound_ = false; +} + +NativeParserModuleInstance& NativeParserModuleInstance::operator=(NativeParserModuleInstance&& other) noexcept { + if (this != &other) { + reset(); + module_ = std::move(other.module_); + token_ = std::exchange(other.token_, PJ_MODULE_CREATION_ERROR_TOKEN); + claim_index_ = other.claim_index_; + bound_route_ = other.bound_route_; + expected_object_type_ = other.expected_object_type_; + bound_ = other.bound_; + other.bound_ = false; + } + return *this; +} + +Expected NativeParserModuleInstance::create( + const NativeParserModule& module, uint32_t claim_index) { + if (!module.valid()) { + return unexpected("cannot create an instance from an invalid native parser module"); + } + const uint64_t token = module.state_->create(claim_index); + if (token == PJ_MODULE_CREATION_ERROR_TOKEN) { + auto message = copyLastError(*module.state_, PJ_MODULE_CREATION_ERROR_TOKEN); + return unexpected(message ? *message : message.error()); + } + return NativeParserModuleInstance(module.state_, token, claim_index); +} + +Expected NativeParserModuleInstance::bind(const parser_module::BindingInfoV1& info) { + if (!valid()) { + return unexpected("cannot bind an invalid native parser-module instance"); + } + if (info.claim_index != claim_index_) { + return unexpected("BindingInfo claim_index does not match the created instance"); + } + auto encoded = parser_module::writeBindingInfoV1(info); + if (!encoded) { + return unexpected(encoded.error()); + } + const int32_t code = module_->bind(token_, addressOf(encoded->data()), encoded->size()); + + ParserModuleBindResult result{ + .outcome = ParserModuleBindOutcome::kError, + .fault = ParserModuleFaultKind::kNone, + .result_code = code, + .message = {}, + }; + if (code == PJ_MODULE_OK) { + result.outcome = ParserModuleBindOutcome::kAccept; + bound_route_ = info.route; + expected_object_type_ = info.expected_object_type; + bound_ = true; + return result; + } + + bound_ = false; + if (code == PJ_MODULE_DECLINE) { + result.outcome = ParserModuleBindOutcome::kDecline; + } else if (code < 0) { + result.outcome = ParserModuleBindOutcome::kError; + if (code == PJ_MODULE_ERR_BAD_TOKEN) { + result.fault = ParserModuleFaultKind::kContractViolation; + } + } else { + result.outcome = ParserModuleBindOutcome::kError; + result.fault = ParserModuleFaultKind::kContractViolation; + result.message = "pj_module_bind returned an out-of-contract positive result"; + return result; + } + + auto message = copyLastError(*module_, token_); + if (!message) { + result.fault = ParserModuleFaultKind::kContractViolation; + result.message = message.error(); + } else { + result.message = std::move(*message); + } + return result; +} + +Expected NativeParserModuleInstance::parse(const parser_module::ParseInputV1& input) { + if (!valid()) { + return unexpected("cannot parse with an invalid native parser-module instance"); + } + if (!bound_) { + return unexpected("cannot parse before an accepted module bind"); + } + auto encoded = parser_module::writeParseInputV1(input); + if (!encoded) { + return unexpected(encoded.error()); + } + + uint64_t output_address = 0; + uint64_t output_length = 0; + const int32_t code = module_->parse( + token_, addressOf(encoded->data()), encoded->size(), addressOf(&output_address), addressOf(&output_length)); + if (code < 0) { + auto message = copyLastError(*module_, token_); + if (!message) { + return contractViolation(code, message.error()); + } + return ParserModuleParseResult{ + .fault = code == PJ_MODULE_ERR_BAD_TOKEN ? ParserModuleFaultKind::kContractViolation + : ParserModuleFaultKind::kDataError, + .result_code = code, + .message = std::move(*message), + .output = std::nullopt, + }; + } + if (code != PJ_MODULE_OK) { + return contractViolation(code, "pj_module_parse returned a nonzero non-error result"); + } + if (output_address == 0 || output_length == 0 || output_length > std::numeric_limits::max()) { + return contractViolation(code, "pj_module_parse returned an unreadable output descriptor"); + } + + const auto* output_bytes = reinterpret_cast(static_cast(output_address)); + auto descriptor = + parser_module::readOutputDescriptorV1(Span(output_bytes, static_cast(output_length))); + if (!descriptor) { + return contractViolation(code, "malformed output descriptor: " + descriptor.error()); + } + + if (bound_route_ == parser_module::Route::kScalar) { + const auto* scalar = std::get_if(&*descriptor); + if (scalar == nullptr) { + return contractViolation(code, "output descriptor route does not match the scalar binding"); + } + auto owned = ownScalarOutput(*scalar); + if (!owned) { + return contractViolation(code, owned.error()); + } + return ParserModuleParseResult{ + .fault = ParserModuleFaultKind::kNone, + .result_code = code, + .message = {}, + .output = ParserModuleOutput(std::move(*owned)), + }; + } + + const auto* object = std::get_if(&*descriptor); + if (object == nullptr) { + return contractViolation(code, "output descriptor route does not match the object binding"); + } + auto owned = ownObjectOutput(*object, input.payload, expected_object_type_); + if (!owned) { + return contractViolation(code, owned.error()); + } + return ParserModuleParseResult{ + .fault = ParserModuleFaultKind::kNone, + .result_code = code, + .message = {}, + .output = ParserModuleOutput(std::move(*owned)), + }; +} + +void NativeParserModuleInstance::reset() noexcept { + if (module_ != nullptr && token_ != PJ_MODULE_CREATION_ERROR_TOKEN) { + module_->destroy(token_); + } + token_ = PJ_MODULE_CREATION_ERROR_TOKEN; + bound_ = false; + module_.reset(); +} + +ParserModuleStrikeState ParserModuleStrikeTracker::recordFault( + const ParserModuleClaimKey& key, ParserModuleFaultKind fault) { + auto [it, inserted] = states_.try_emplace(key); + (void)inserted; + auto& state = it->second; + if (fault != ParserModuleFaultKind::kContractViolation || state.health != ParserModuleClaimHealth::kActive) { + return state; + } + + ++state.strikes; + if (state.strikes == 3) { + state.strikes = 0; + ++state.quarantine_count; + if (state.quarantine_count == 1) { + state.health = ParserModuleClaimHealth::kQuarantined; + } else { + state.health = ParserModuleClaimHealth::kDisabled; + } + } + return state; +} + +bool ParserModuleStrikeTracker::markRecreated(const ParserModuleClaimKey& key) { + auto it = states_.find(key); + if (it == states_.end() || it->second.health != ParserModuleClaimHealth::kQuarantined) { + return false; + } + it->second.health = ParserModuleClaimHealth::kActive; + return true; +} + +ParserModuleStrikeState ParserModuleStrikeTracker::state(const ParserModuleClaimKey& key) const { + const auto it = states_.find(key); + return it == states_.end() ? ParserModuleStrikeState{} : it->second; +} + +} // namespace PJ diff --git a/pj_plugins/src/parser_route_resolver.cpp b/pj_plugins/src/parser_route_resolver.cpp new file mode 100644 index 00000000..1b4efa65 --- /dev/null +++ b/pj_plugins/src/parser_route_resolver.cpp @@ -0,0 +1,362 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_route_resolver.hpp" + +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_route_claims_protocol.h" + +namespace PJ { +namespace { + +[[nodiscard]] uint16_t routeFlag(ParserRoute route) noexcept { + return route == ParserRoute::kScalar ? PJ_PARSER_ROUTE_FLAG_SCALAR_V1 : PJ_PARSER_ROUTE_FLAG_OBJECT_V1; +} + +[[nodiscard]] ParserClaimIdentity identityOf(const ParserClaim& claim) { + return {claim.provider_id, claim.claim_id}; +} + +[[nodiscard]] bool hasSamePolicyRank(const ParserClaim& lhs, const ParserClaim& rhs) noexcept { + return (lhs.type_name == "*") == (rhs.type_name == "*") && lhs.provenance == rhs.provenance && + lhs.priority == rhs.priority; +} + +[[nodiscard]] std::string outcomeDetail(const ParserProbeDecision& decision, std::string_view fallback) { + return decision.diagnostic.empty() ? std::string(fallback) : decision.diagnostic; +} + +} // namespace + +struct ParserRouteResolver::CacheEntry { + std::string provider_id; + std::string claim_id; + uint64_t provider_generation = 0; + std::string encoding; + std::string type_name; + std::string schema_digest; + std::string config_digest; + ParserProbeDecision decision; +}; + +ParserRouteResolver::ParserRouteResolver(DiagnosticSink sink, std::string diagnostic_source) + : sink_(std::move(sink)), diagnostic_source_(std::move(diagnostic_source)) {} + +ParserRouteResolver::~ParserRouteResolver() = default; + +void ParserRouteResolver::setDiagnosticSink(DiagnosticSink sink) { + sink_ = std::move(sink); +} + +Expected> ParserRouteResolver::orderedCandidates( + const ParserRouteRequest& request, const ParserClaimCatalog& catalog, const ParserRoutePins& pins) const { + if (!isRegisteredParserEncoding(request.encoding)) { + return unexpected("route request uses unknown encoding: " + request.encoding); + } + auto normalized_type = normalizeParserTypeName(request.encoding, request.type_name); + if (!normalized_type) { + return unexpected(normalized_type.error()); + } + + const auto& pin = pins.forRoute(request.route); + std::vector candidates; + for (const auto& entry : catalog.claims()) { + const ParserClaim& claim = entry.claim; + if (claim.encoding != request.encoding || (claim.route_flags & routeFlag(request.route)) == 0) { + continue; + } + if (claim.type_name != "*" && claim.type_name != *normalized_type) { + continue; + } + if (!claim.schema_digests.empty() && !claim.schema_digests.contains(request.schema_digest)) { + continue; + } + if (pin.has_value() && claim.provider_id != *pin) { + continue; + } + candidates.push_back({.claim = claim, .provider_generation = entry.provider_generation}); + } + + std::sort(candidates.begin(), candidates.end(), [](const ParserRouteCandidate& lhs, const ParserRouteCandidate& rhs) { + const bool lhs_exact = lhs.claim.type_name != "*"; + const bool rhs_exact = rhs.claim.type_name != "*"; + if (lhs_exact != rhs_exact) { + return lhs_exact; + } + if (lhs.claim.provenance != rhs.claim.provenance) { + return lhs.claim.provenance > rhs.claim.provenance; + } + if (lhs.claim.priority != rhs.claim.priority) { + return lhs.claim.priority > rhs.claim.priority; + } + return std::tie(lhs.claim.provider_id, lhs.claim.claim_id) < std::tie(rhs.claim.provider_id, rhs.claim.claim_id); + }); + return candidates; +} + +Expected ParserRouteResolver::resolve( + const ParserRouteRequest& request, const ParserClaimCatalog& catalog, const ParserRoutePins& pins, + const ParserProviderConfigLookup& provider_config, const ParserProbeCallback& probe) { + if (!probe) { + return unexpected("parser probe callback is empty"); + } + auto normalized_type = normalizeParserTypeName(request.encoding, request.type_name); + if (!normalized_type) { + return unexpected(normalized_type.error()); + } + auto candidates = orderedCandidates(request, catalog, pins); + if (!candidates) { + return unexpected(candidates.error()); + } + + ParserRouteResolution resolution; + resolution.ordered_candidates.reserve(candidates->size()); + resolution.trace.reserve(candidates->size() * 3 + 2); + for (const auto& candidate : *candidates) { + const auto identity = identityOf(candidate.claim); + resolution.ordered_candidates.push_back(identity); + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kCandidate, + .claim = identity, + .detail = candidate.claim.type_name == "*" ? "wildcard candidate" : "exact candidate", + }); + } + + const auto& pin = pins.forRoute(request.route); + if (candidates->empty()) { + if (pin.has_value()) { + resolution.status = ParserRouteResolutionStatus::kPinnedProviderUnavailable; + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kPinnedProviderUnavailable, + .claim = {.provider_id = *pin, .claim_id = {}}, + .detail = "pinned provider has no matching admitted claim", + }); + report( + DiagnosticLevel::kError, *pin, + "pinned parser provider is unavailable for " + request.encoding + ":" + *normalized_type); + } else { + resolution.status = ParserRouteResolutionStatus::kNoCandidates; + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kExhausted, + .claim = {}, + .detail = "no matching admitted claims", + }); + } + return resolution; + } + + std::vector declined_probes; + const auto report_declines = [&]() { + if (declined_probes.empty()) { + return; + } + std::string summary = "parser probe declined for " + std::to_string(declined_probes.size()) + " candidate(s): "; + for (size_t index = 0; index < declined_probes.size(); ++index) { + if (index != 0) { + summary += "; "; + } + summary += declined_probes[index]; + } + report(DiagnosticLevel::kInfo, {}, std::move(summary)); + }; + + for (const auto& candidate : *candidates) { + ParserProviderConfig config; + if (provider_config) { + try { + config = provider_config(candidate.claim.provider_id); + } catch (const std::exception& error) { + return unexpected( + "provider config lookup failed for provider " + candidate.claim.provider_id + ": " + error.what()); + } catch (...) { + return unexpected("provider config lookup failed for provider " + candidate.claim.provider_id); + } + } + + auto& route_cache = request.route == ParserRoute::kScalar ? scalar_probe_cache_ : object_probe_cache_; + auto cached = std::find_if(route_cache.begin(), route_cache.end(), [&](const CacheEntry& entry) { + return entry.provider_id == candidate.claim.provider_id && entry.claim_id == candidate.claim.claim_id && + entry.provider_generation == candidate.provider_generation && entry.encoding == request.encoding && + entry.type_name == *normalized_type && entry.schema_digest == request.schema_digest && + entry.config_digest == config.digest; + }); + + ParserProbeDecision decision; + const bool cache_hit = cached != route_cache.end(); + if (cache_hit) { + decision = cached->decision; + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kCacheHit, + .claim = identityOf(candidate.claim), + .detail = "probe result reused", + }); + } else { + try { + decision = probe( + ParserProbeRequest{ + .claim = candidate.claim, + .provider_generation = candidate.provider_generation, + .config_json = config.json, + .config_digest = config.digest, + }); + } catch (const std::exception& error) { + decision = { + .outcome = ParserProbeOutcome::kError, + .diagnostic = std::string("probe callback threw: ") + error.what(), + .retained_instance = {}, + }; + } catch (...) { + decision = { + .outcome = ParserProbeOutcome::kError, + .diagnostic = "probe callback threw an unknown exception", + .retained_instance = {}, + }; + } + if (decision.outcome != ParserProbeOutcome::kAccept && decision.outcome != ParserProbeOutcome::kDecline && + decision.outcome != ParserProbeOutcome::kError) { + decision = { + .outcome = ParserProbeOutcome::kError, + .diagnostic = "probe callback returned an invalid outcome", + .retained_instance = {}, + }; + } else if (decision.outcome != ParserProbeOutcome::kAccept) { + decision.retained_instance.reset(); + } + route_cache.push_back( + CacheEntry{ + .provider_id = candidate.claim.provider_id, + .claim_id = candidate.claim.claim_id, + .provider_generation = candidate.provider_generation, + .encoding = request.encoding, + .type_name = *normalized_type, + .schema_digest = request.schema_digest, + .config_digest = config.digest, + .decision = decision, + }); + } + + if (decision.outcome == ParserProbeOutcome::kDecline) { + const std::string detail = outcomeDetail(decision, "provider declined during probe"); + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kProbeDecline, + .claim = identityOf(candidate.claim), + .detail = detail, + }); + if (!cache_hit) { + declined_probes.push_back(candidate.claim.provider_id + "/" + candidate.claim.claim_id + ": " + detail); + } + continue; + } + if (decision.outcome == ParserProbeOutcome::kError) { + const std::string detail = outcomeDetail(decision, "provider probe failed"); + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kProbeError, + .claim = identityOf(candidate.claim), + .detail = detail, + }); + if (!cache_hit) { + report(DiagnosticLevel::kError, candidate.claim.provider_id, "parser probe error: " + detail); + } + continue; + } + + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kProbeAccept, + .claim = identityOf(candidate.claim), + .detail = "provider accepted during probe", + }); + resolution.status = ParserRouteResolutionStatus::kSelected; + resolution.winner = identityOf(candidate.claim); + resolution.winning_claim = candidate.claim; + resolution.retained_instance = std::move(decision.retained_instance); + report_declines(); + + const auto tied = std::find_if(candidates->begin(), candidates->end(), [&](const ParserRouteCandidate& other) { + return identityOf(other.claim) > *resolution.winner && hasSamePolicyRank(candidate.claim, other.claim); + }); + if (tied != candidates->end()) { + std::string ambiguity_key = request.encoding + "\n" + *normalized_type + "\n" + request.schema_digest + "\n" + + std::to_string(static_cast(request.route)) + "\n" + + resolution.winner->provider_id + "\n" + resolution.winner->claim_id; + if (std::find(reported_ambiguities_.begin(), reported_ambiguities_.end(), ambiguity_key) == + reported_ambiguities_.end()) { + reported_ambiguities_.push_back(std::move(ambiguity_key)); + const std::string detail = "stable claim identity selected " + resolution.winner->provider_id + "/" + + resolution.winner->claim_id + " over an equal-ranked candidate"; + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kAmbiguityTieBreak, + .claim = *resolution.winner, + .detail = detail, + }); + report(DiagnosticLevel::kWarning, resolution.winner->provider_id, "ambiguous parser claims: " + detail); + } + } + + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kSelected, + .claim = *resolution.winner, + .detail = "route provider selected", + }); + return resolution; + } + + resolution.status = pin.has_value() ? ParserRouteResolutionStatus::kPinnedProviderRejected + : ParserRouteResolutionStatus::kNoProviderAccepted; + resolution.trace.push_back({ + .kind = ParserSelectionTraceKind::kExhausted, + .claim = {.provider_id = pin.value_or(""), .claim_id = {}}, + .detail = pin.has_value() ? "pinned provider declined or failed; fallback is disabled" + : "all matching providers declined or failed", + }); + report_declines(); + if (pin.has_value()) { + report(DiagnosticLevel::kError, *pin, "pinned parser provider declined or failed; route remains unbound"); + } + return resolution; +} + +void ParserRouteResolver::invalidateCatalog() { + clearAllCachedState(); +} + +void ParserRouteResolver::invalidatePins() { + clearAllCachedState(); +} + +void ParserRouteResolver::invalidateProviderConfig(std::string_view provider_id) { + const auto belongs_to_provider = [&](const CacheEntry& entry) { return entry.provider_id == provider_id; }; + std::erase_if(scalar_probe_cache_, belongs_to_provider); + std::erase_if(object_probe_cache_, belongs_to_provider); +} + +size_t ParserRouteResolver::probeCacheSize() const noexcept { + return scalar_probe_cache_.size() + object_probe_cache_.size(); +} + +void ParserRouteResolver::clearAllCachedState() { + scalar_probe_cache_.clear(); + object_probe_cache_.clear(); + reported_ambiguities_.clear(); +} + +void ParserRouteResolver::report(DiagnosticLevel level, std::string_view id, std::string message) const { + if (!sink_) { + return; + } + sink_( + Diagnostic{ + .level = level, + .source = diagnostic_source_, + .id = std::string(id), + .message = std::move(message), + .timestamp = std::chrono::system_clock::now(), + }); +} + +} // namespace PJ diff --git a/pj_plugins/tests/message_parser_functional_extension_test.cpp b/pj_plugins/tests/message_parser_functional_extension_test.cpp index 3f34066b..ba6c638a 100644 --- a/pj_plugins/tests/message_parser_functional_extension_test.cpp +++ b/pj_plugins/tests/message_parser_functional_extension_test.cpp @@ -4,6 +4,7 @@ #include #include +#include #include #include #include @@ -161,6 +162,32 @@ bool emitUnknownObject( return sink->accept_object(sink->ctx, false, 0, 999, PJ_bytes_view_t{&byte, sizeof(byte)}, out_error); } +bool emitImageV1(void*, int64_t, PJ_payload_t, const PJ_parser_object_sink_v1_t* sink, PJ_error_t* out_error) noexcept { + static constexpr std::array kImageWire{0x10, 0x01}; + return sink != nullptr && sink->accept_object != nullptr && + sink->accept_object( + sink->ctx, false, 0, PJ_BUILTIN_OBJECT_TYPE_IMAGE, PJ_bytes_view_t{kImageWire.data(), kImageWire.size()}, + out_error); +} + +bool emitSplicedPointCloudV2( + void*, int64_t, PJ_payload_t, const PJ_parser_object_sink_v2_t* sink, PJ_error_t* out_error) noexcept { + static constexpr std::array kPointCloudWire{0x10, 0x01}; + return sink != nullptr && sink->accept_object_spliced != nullptr && + sink->accept_object_spliced( + sink->ctx, true, 88, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + PJ_bytes_view_t{kPointCloudWire.data(), kPointCloudWire.size()}, 9, 1, 2, out_error); +} + +bool emitMismatchedImageV2( + void*, int64_t, PJ_payload_t, const PJ_parser_object_sink_v2_t* sink, PJ_error_t* out_error) noexcept { + static constexpr std::array kImageWire{0x10, 0x01}; + return sink != nullptr && sink->accept_object != nullptr && + sink->accept_object( + sink->ctx, false, 0, PJ_BUILTIN_OBJECT_TYPE_IMAGE, PJ_bytes_view_t{kImageWire.data(), kImageWire.size()}, + out_error); +} + template const PJ_message_parser_vtable_t* adversarialVtable() { static const PJ_parser_functional_v1_t extension{ @@ -179,6 +206,33 @@ const PJ_message_parser_vtable_t* adversarialVtable() { return &vtable; } +template +const PJ_message_parser_vtable_t* adversarialV2Vtable() { + static const PJ_parser_functional_v2_t extension{ + .struct_size = sizeof(PJ_parser_functional_v2_t), + .parse_scalars = emitNoScalars, + .parse_object = ParseObject, + }; + static const PJ_message_parser_vtable_t vtable = [] { + PJ_message_parser_vtable_t copy = *parserVtable(); + copy.classify_schema = [](void*, PJ_string_view_t, PJ_bytes_view_t, PJ_schema_classification_t* out, + PJ_error_t*) noexcept { + if (out == nullptr) { + return false; + } + out->object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD; + out->reserved = 0; + return true; + }; + copy.get_plugin_extension = [](void*, PJ_string_view_t id) noexcept -> const void* { + const std::string_view name{id.data == nullptr ? "" : id.data, id.size}; + return name == PJ_PARSER_FUNCTIONAL_EXTENSION_V2 ? &extension : nullptr; + }; + return copy; + }(); + return &vtable; +} + TEST(MessageParserFunctionalExtension, NewlyBuiltParserExposesStableExtensionAutomatically) { PJ::MessageParserHandle handle(parserVtable()); @@ -190,6 +244,129 @@ TEST(MessageParserFunctionalExtension, NewlyBuiltParserExposesStableExtensionAut EXPECT_GE(extension->struct_size, PJ_PARSER_FUNCTIONAL_V1_MIN_SIZE); } +TEST(MessageParserFunctionalExtension, NewlyBuiltParserExposesV1AndV2AndV2EmitsFullCanonicalWire) { + PJ::MessageParserHandle handle(parserVtable()); + ASSERT_TRUE(handle.bindSchema(kSchema, {})); + + const auto* v1 = + static_cast(handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V1)); + const auto* v2 = + static_cast(handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2)); + ASSERT_NE(v1, nullptr); + ASSERT_NE(v2, nullptr); + EXPECT_GE(v1->struct_size, PJ_PARSER_FUNCTIONAL_V1_MIN_SIZE); + EXPECT_GE(v2->struct_size, PJ_PARSER_FUNCTIONAL_V2_MIN_SIZE); + EXPECT_NE(v2->parse_scalars, nullptr); + + struct SinkState { + bool accepted_full = false; + bool accepted_spliced = false; + uint16_t object_type = PJ_BUILTIN_OBJECT_TYPE_NONE; + } state; + PJ_parser_object_sink_v2_t sink{ + .struct_size = sizeof(PJ_parser_object_sink_v2_t), + .ctx = &state, + .accept_object = + [](void* ctx, bool, int64_t, uint16_t object_type, PJ_bytes_view_t wire, PJ_error_t*) noexcept { + auto& captured = *static_cast(ctx); + captured.accepted_full = true; + captured.object_type = object_type; + return wire.size != 0; + }, + .accept_object_spliced = + [](void* ctx, bool, int64_t, uint16_t, PJ_bytes_view_t, uint32_t, uint64_t, uint64_t, PJ_error_t*) noexcept { + static_cast(ctx)->accepted_spliced = true; + return true; + }, + }; + const std::array payload{1, 2, 3}; + PJ_error_t error{}; + EXPECT_TRUE(v2->parse_object( + handle.context(), 100, PJ_payload_t{.data = payload.data(), .size = payload.size(), .anchor = {}}, &sink, &error)) + << error.message; + EXPECT_TRUE(state.accepted_full); + EXPECT_FALSE(state.accepted_spliced); + EXPECT_EQ(state.object_type, PJ_BUILTIN_OBJECT_TYPE_IMAGE); +} + +TEST(MessageParserFunctionalExtension, V2FailureKindsAreFrozenAndCarryNoExtendedPayload) { + PJ::MessageParserHandle throwing_handle(parserVtable()); + ASSERT_TRUE(throwing_handle.bindSchema(kSchema, {})); + const auto* throwing_v2 = static_cast( + throwing_handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2)); + ASSERT_NE(throwing_v2, nullptr); + + PJ_parser_scalar_sink_v1_t scalar_sink{ + .struct_size = sizeof(PJ_parser_scalar_sink_v1_t), + .ctx = nullptr, + .accept_record = [](void*, bool, int64_t, const PJ_named_field_value_t*, uint64_t, + PJ_error_t*) noexcept { return true; }, + }; + PJ_error_t error{}; + EXPECT_FALSE(throwing_v2->parse_scalars(throwing_handle.context(), 0, PJ_bytes_view_t{}, &scalar_sink, &error)); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + EXPECT_EQ(error.extended, nullptr); + + PJ_parser_object_sink_v2_t object_sink{ + .struct_size = sizeof(PJ_parser_object_sink_v2_t), + .ctx = nullptr, + .accept_object = [](void*, bool, int64_t, uint16_t, PJ_bytes_view_t, PJ_error_t*) noexcept { return true; }, + .accept_object_spliced = [](void*, bool, int64_t, uint16_t, PJ_bytes_view_t, uint32_t, uint64_t, uint64_t, + PJ_error_t*) noexcept { return true; }, + }; + error = {}; + EXPECT_FALSE(throwing_v2->parse_object(throwing_handle.context(), 0, PJ_payload_t{}, &object_sink, &error)); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + EXPECT_EQ(error.extended, nullptr); + + PJ::MessageParserHandle handle(parserVtable()); + ASSERT_TRUE(handle.bindSchema(kSchema, {})); + const auto* v2 = + static_cast(handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2)); + ASSERT_NE(v2, nullptr); + scalar_sink.accept_record = [](void*, bool, int64_t, const PJ_named_field_value_t*, uint64_t, + PJ_error_t* out_error) noexcept { + PJ::sdk::fillError(out_error, 7, "test", "sink declined record"); + return false; + }; + error = {}; + EXPECT_FALSE(v2->parse_scalars(handle.context(), 0, PJ_bytes_view_t{}, &scalar_sink, &error)); + EXPECT_EQ(std::string_view(error.message), "sink declined record"); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_SINK_REJECTED); + EXPECT_EQ(error.extended, nullptr); + + scalar_sink.accept_record = [](void*, bool, int64_t, const PJ_named_field_value_t*, uint64_t, + PJ_error_t* out_error) noexcept { + PJ::sdk::fillError(out_error, 7, "test", "sink detected a contract violation"); + PJ::sdk::setExtended(out_error, PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION, nullptr); + return false; + }; + error = {}; + EXPECT_FALSE(v2->parse_scalars(handle.context(), 0, PJ_bytes_view_t{}, &scalar_sink, &error)); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + + object_sink.accept_object = [](void*, bool, int64_t, uint16_t, PJ_bytes_view_t, PJ_error_t* out_error) noexcept { + PJ::sdk::fillError(out_error, 7, "test", "object type contract violation"); + PJ::sdk::setExtended(out_error, PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION, nullptr); + return false; + }; + error = {}; + EXPECT_FALSE(v2->parse_object(handle.context(), 0, PJ_payload_t{}, &object_sink, &error)); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_CONTRACT_VIOLATION); + + PJ::MessageParserHandle unbound_handle(parserVtable()); + const auto* unbound_v2 = static_cast( + unbound_handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2)); + ASSERT_NE(unbound_v2, nullptr); + scalar_sink.accept_record = [](void*, bool, int64_t, const PJ_named_field_value_t*, uint64_t, PJ_error_t*) noexcept { + return true; + }; + error = {}; + EXPECT_FALSE(unbound_v2->parse_scalars(unbound_handle.context(), 0, PJ_bytes_view_t{}, &scalar_sink, &error)); + EXPECT_EQ(std::string_view(error.extended_kind), PJ_PARSER_ERROR_KIND_DATA_ERROR); + EXPECT_EQ(error.extended, nullptr); +} + TEST(MessageParserFunctionalExtension, ScalarRouteDeliversOnlyCAbiValuesDuringTheSinkCall) { PJ::MessageParserHandle handle(parserVtable()); ASSERT_TRUE(handle.bindSchema(kSchema, {})); @@ -246,6 +423,36 @@ TEST(MessageParserFunctionalExtension, ObjectRouteReturnsAHostOwnedTypedValue) { EXPECT_EQ(image->data[3], 6U); } +TEST(MessageParserFunctionalExtension, HostFallsBackToV1WhenV2IsAbsent) { + PJ::MessageParserHandle handle(adversarialVtable()); + ASSERT_TRUE(handle.bindSchema(kSchema, {})); + auto record = handle.parseObjectFunctional(0, PJ::Span{}); + ASSERT_TRUE(record.has_value()) << record.error(); + EXPECT_EQ(PJ::sdk::typeOf(record->object), PJ::sdk::BuiltinObjectType::kImage); +} + +TEST(MessageParserFunctionalExtension, HostV2PathReconstructsEligibleSplices) { + PJ::MessageParserHandle handle(adversarialV2Vtable()); + ASSERT_TRUE(handle.bindSchema("example/PointCloud", {})); + const std::array payload{10, 20, 30, 40}; + auto record = handle.parseObjectFunctional(0, PJ::Span(payload)); + ASSERT_TRUE(record.has_value()) << record.error(); + EXPECT_EQ(record->ts, 88); + const auto* cloud = std::any_cast(&record->object); + ASSERT_NE(cloud, nullptr); + ASSERT_EQ(cloud->data.size(), 2U); + EXPECT_EQ(cloud->data[0], 20U); + EXPECT_EQ(cloud->data[1], 30U); +} + +TEST(MessageParserFunctionalExtension, HostRejectsObjectTypeThatDiffersFromBindingClassification) { + PJ::MessageParserHandle handle(adversarialV2Vtable()); + ASSERT_TRUE(handle.bindSchema("example/PointCloud", {})); + auto record = handle.parseObjectFunctional(0, PJ::Span{}); + ASSERT_FALSE(record.has_value()); + EXPECT_NE(record.error().find("differs from the bound classification"), std::string::npos); +} + TEST(MessageParserFunctionalExtension, PluginExceptionsNeverCrossEitherFunctionalRoute) { PJ::MessageParserHandle handle(parserVtable()); ASSERT_TRUE(handle.bindSchema(kSchema, {})); @@ -269,6 +476,7 @@ TEST(MessageParserFunctionalExtension, ExistingPluginDefinedExtensionsRemainVisi PJ::MessageParserHandle handle(parserVtable()); EXPECT_FALSE(handle.supportsFunctionalParsing()); + EXPECT_EQ(handle.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2), nullptr); const auto* marker = static_cast(handle.getPluginExtension("example.custom.v1")); ASSERT_NE(marker, nullptr); EXPECT_EQ(*marker, 17); @@ -318,6 +526,21 @@ TEST(MessageParserFunctionalExtension, RebindingToAnUnhandledSchemaWithdrawsTheF EXPECT_FALSE(handle.supportsFunctionalParsing()); } +TEST(MessageParserFunctionalExtension, SchemaGatingWithdrawsV2AlongsideV1) { + // v2 shares v1's advertisement gate. A host prefers v2, so a v2 that stayed + // advertised for a schema only legacy parse() implements would send every + // message down a route parse_object/parse_scalars can only reject. + PJ::MessageParserHandle handled(parserVtable()); + ASSERT_TRUE(handled.bindSchema(kSchema, {})); + EXPECT_NE(handled.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V1), nullptr); + EXPECT_NE(handled.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2), nullptr); + + PJ::MessageParserHandle unhandled(parserVtable()); + ASSERT_TRUE(unhandled.bindSchema("example/Unhandled", {})); + EXPECT_EQ(unhandled.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V1), nullptr); + EXPECT_EQ(unhandled.getPluginExtension(PJ_PARSER_FUNCTIONAL_EXTENSION_V2), nullptr); +} + TEST(MessageParserFunctionalExtension, LegacyParserWithoutExtensionRemainsDetectable) { PJ::MessageParserHandle handle(legacyStyleVtable()); diff --git a/pj_plugins/tests/message_parser_route_claims_extension_test.cpp b/pj_plugins/tests/message_parser_route_claims_extension_test.cpp new file mode 100644 index 00000000..451874b1 --- /dev/null +++ b/pj_plugins/tests/message_parser_route_claims_extension_test.cpp @@ -0,0 +1,145 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +#include +#include +#include + +#include "pj_base/parser_route_claims_protocol.h" +#include "pj_plugins/host/message_parser_handle.hpp" +#include "pj_plugins/sdk/message_parser_plugin_base.hpp" + +namespace { + +class RouteClaimsParser final : public PJ::MessageParserPluginBase { + public: + RouteClaimsParser() { + PJ::sdk::SchemaHandler both; + both.object_type = PJ::sdk::BuiltinObjectType::kImage; + both.parse_scalars = [](PJ::Timestamp, PJ::Span) -> PJ::Expected { + return PJ::sdk::ScalarRecord{}; + }; + both.parse_object = [](PJ::Timestamp, PJ::sdk::PayloadView) -> PJ::Expected { + return PJ::unexpected("not invoked by classification"); + }; + registerSchemaHandler("example/Both", std::move(both)); + + PJ::sdk::SchemaHandler scalar; + scalar.object_type = PJ::sdk::BuiltinObjectType::kPointCloud; + scalar.parse_scalars = [](PJ::Timestamp, PJ::Span) -> PJ::Expected { + return PJ::sdk::ScalarRecord{}; + }; + registerSchemaHandler("example/Scalar", std::move(scalar)); + + PJ::sdk::SchemaHandler object; + object.object_type = PJ::sdk::BuiltinObjectType::kDepthImage; + object.parse_object = [](PJ::Timestamp, PJ::sdk::PayloadView) -> PJ::Expected { + return PJ::unexpected("not invoked by classification"); + }; + registerSchemaHandler("example/Object", std::move(object)); + } +}; + +class EmptyParser final : public PJ::MessageParserPluginBase {}; + +template +const PJ_message_parser_vtable_t* parserVtable() { + static const auto* vtable = PJ::MessageParserPluginBase::vtableWithCreate( + []() noexcept -> void* { + try { + return new Parser(); + } catch (...) { + return nullptr; + } + }, + R"({"id":"route-claims-test","name":"Route Claims Test","version":"1.0.0","encoding":["test"]})"); + return vtable; +} + +const PJ_parser_route_claims_v1_t* claimsExtension(PJ::MessageParserHandle& handle) { + return static_cast( + handle.getPluginExtension(PJ_PARSER_ROUTE_CLAIMS_EXTENSION_V1)); +} + +PJ_route_classification_v1_t classify( + PJ::MessageParserHandle& handle, const PJ_parser_route_claims_v1_t& extension, std::string_view type_name) { + PJ_route_classification_v1_t result{}; + PJ_error_t error{}; + const PJ_string_view_t name{type_name.data(), type_name.size()}; + EXPECT_TRUE(extension.classify_routes(handle.context(), name, PJ_bytes_view_t{}, &result, &error)) << error.message; + return result; +} + +TEST(MessageParserRouteClaimsExtension, RegisteredHandlersProduceExactPerRouteClaims) { + PJ::MessageParserHandle handle(parserVtable()); + const auto* extension = claimsExtension(handle); + ASSERT_NE(extension, nullptr); + ASSERT_GE(extension->struct_size, PJ_PARSER_ROUTE_CLAIMS_V1_MIN_SIZE); + + const auto both = classify(handle, *extension, "example/Both"); + EXPECT_EQ(both.route_flags, PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1); + EXPECT_EQ(both.match, PJ_PARSER_ROUTE_MATCH_EXACT_V1); + EXPECT_EQ(both.status, PJ_PARSER_ROUTE_STATUS_CLAIMED_V1); + EXPECT_EQ(both.object_type, PJ_BUILTIN_OBJECT_TYPE_IMAGE); + + const auto scalar = classify(handle, *extension, "example/Scalar"); + EXPECT_EQ(scalar.route_flags, PJ_PARSER_ROUTE_FLAG_SCALAR_V1); + EXPECT_EQ(scalar.match, PJ_PARSER_ROUTE_MATCH_EXACT_V1); + EXPECT_EQ(scalar.status, PJ_PARSER_ROUTE_STATUS_CLAIMED_V1); + EXPECT_EQ(scalar.object_type, PJ_BUILTIN_OBJECT_TYPE_NONE); + + const auto object = classify(handle, *extension, "example/Object"); + EXPECT_EQ(object.route_flags, PJ_PARSER_ROUTE_FLAG_OBJECT_V1); + EXPECT_EQ(object.match, PJ_PARSER_ROUTE_MATCH_EXACT_V1); + EXPECT_EQ(object.status, PJ_PARSER_ROUTE_STATUS_CLAIMED_V1); + EXPECT_EQ(object.object_type, PJ_BUILTIN_OBJECT_TYPE_DEPTH_IMAGE); +} + +TEST(MessageParserRouteClaimsExtension, UnknownTypeDeclinesWithoutExpressingWildcardCoverage) { + PJ::MessageParserHandle handle(parserVtable()); + const auto* extension = claimsExtension(handle); + ASSERT_NE(extension, nullptr); + + const auto result = classify(handle, *extension, "example/Unknown"); + EXPECT_EQ(result.route_flags, 0); + EXPECT_EQ(result.match, PJ_PARSER_ROUTE_MATCH_EXACT_V1); + EXPECT_EQ(result.status, PJ_PARSER_ROUTE_STATUS_DECLINED_V1); + EXPECT_EQ(result.object_type, PJ_BUILTIN_OBJECT_TYPE_NONE); +} + +TEST(MessageParserRouteClaimsExtension, RebuiltParserWithoutHandlersStillExposesDecliningClassifier) { + PJ::MessageParserHandle handle(parserVtable()); + const auto* extension = claimsExtension(handle); + ASSERT_NE(extension, nullptr); + + const auto result = classify(handle, *extension, "anything"); + EXPECT_EQ(result.route_flags, 0); + EXPECT_EQ(result.match, PJ_PARSER_ROUTE_MATCH_EXACT_V1); + EXPECT_EQ(result.status, PJ_PARSER_ROUTE_STATUS_DECLINED_V1); +} + +TEST(MessageParserRouteClaimsExtension, InvalidCallsAreFailuresNotDeclineStatuses) { + PJ::MessageParserHandle handle(parserVtable()); + const auto* extension = claimsExtension(handle); + ASSERT_NE(extension, nullptr); + + PJ_error_t error{}; + const PJ_string_view_t name{"example/Both", 12}; + EXPECT_FALSE(extension->classify_routes(handle.context(), name, PJ_bytes_view_t{}, nullptr, &error)); + EXPECT_NE(std::string_view(error.message).find("null output"), std::string_view::npos); + + error = {}; + PJ_route_classification_v1_t result{ + .route_flags = 99, + .match = 99, + .status = 99, + .object_type = 99, + }; + EXPECT_FALSE(extension->classify_routes(handle.context(), name, PJ_bytes_view_t{nullptr, 1}, &result, &error)); + EXPECT_NE(std::string_view(error.message).find("invalid borrowed view"), std::string_view::npos); + EXPECT_EQ(result.status, 99); +} + +} // namespace diff --git a/pj_plugins/tests/native_parser_module_fixture.cpp b/pj_plugins/tests/native_parser_module_fixture.cpp new file mode 100644 index 00000000..612921dc --- /dev/null +++ b/pj_plugins/tests/native_parser_module_fixture.cpp @@ -0,0 +1,294 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "native_parser_module_fixture.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin/point_cloud_codec.hpp" +#include "pj_base/builtin_object_abi.h" +#include "pj_base/parser_module_abi.h" +#include "pj_base/span.hpp" + +#if defined(_WIN32) +#define PJ_FIXTURE_EXPORT __declspec(dllexport) +#else +#define PJ_FIXTURE_EXPORT __attribute__((visibility("default"))) +#endif + +namespace { + +using enum pj_fixture::ClaimIndex; + +constexpr char kManifest[] = R"({ + "module_abi":1, + "id":"org.plotjuggler.test.native-module", + "name":"Native module test fixture", + "version":"1.0.0", + "claims":[ + {"claim_id":"object","encoding":"protobuf","type_name":"fixture.Object","routes":["object"],"object_type":"kPointCloud","priority":0}, + {"claim_id":"scalar","encoding":"protobuf","type_name":"fixture.Scalar","routes":["scalar"],"priority":0}, + {"claim_id":"decline","encoding":"protobuf","type_name":"fixture.Decline","routes":["scalar"],"priority":0}, + {"claim_id":"create-failure","encoding":"protobuf","type_name":"fixture.CreateFailure","routes":["scalar"],"priority":0}, + {"claim_id":"data-error","encoding":"protobuf","type_name":"fixture.DataError","routes":["scalar"],"priority":0}, + {"claim_id":"malformed","encoding":"protobuf","type_name":"fixture.Malformed","routes":["scalar"],"priority":0}, + {"claim_id":"splice","encoding":"protobuf","type_name":"fixture.Splice","routes":["object"],"object_type":"kPointCloud","priority":0}, + {"claim_id":"splice-oob","encoding":"protobuf","type_name":"fixture.SpliceOob","routes":["object"],"object_type":"kPointCloud","priority":0}, + {"claim_id":"splice-ineligible","encoding":"protobuf","type_name":"fixture.SpliceIneligible","routes":["object"],"object_type":"kPointCloud","priority":0}, + {"claim_id":"bad-token","encoding":"protobuf","type_name":"fixture.BadToken","routes":["scalar"],"priority":0}, + {"claim_id":"route-mismatch","encoding":"protobuf","type_name":"fixture.RouteMismatch","routes":["scalar"],"priority":0}, + {"claim_id":"type-mismatch","encoding":"protobuf","type_name":"fixture.TypeMismatch","routes":["object"],"object_type":"kPointCloud","priority":0} + ] +})"; + +struct Instance { + explicit Instance(uint32_t selected_claim) : claim_index(selected_claim) {} + + uint32_t claim_index = 0; + PJ::parser_module::Route route = PJ::parser_module::Route::kScalar; + std::string error; + std::vector output; +}; + +std::unordered_set& liveInstances() { + static std::unordered_set instances; + return instances; +} + +std::string& creationError() { + static std::string error; + return error; +} + +uint64_t addressOf(const void* pointer) { + return static_cast(reinterpret_cast(pointer)); +} + +Instance* findInstance(uint64_t token) { + auto* instance = reinterpret_cast(static_cast(token)); + return liveInstances().contains(instance) ? instance : nullptr; +} + +int32_t fail(Instance* instance, int32_t code, std::string message) { + if (instance != nullptr) { + instance->error = std::move(message); + } + return code; +} + +int32_t storeOutput(Instance& instance, PJ::parser_module::OutputDescriptorV1 descriptor) { + auto output = PJ::parser_module::writeOutputDescriptorV1(descriptor); + if (!output) { + return fail(&instance, PJ_MODULE_ERR_GENERIC, output.error()); + } + instance.output = std::move(*output); + return PJ_MODULE_OK; +} + +} // namespace + +extern "C" { + +PJ_FIXTURE_EXPORT uint32_t pj_module_abi() { +#if defined(PJ_FIXTURE_WRONG_ABI) + return PJ_PARSER_MODULE_ABI_VERSION + 1; +#else + return PJ_PARSER_MODULE_ABI_VERSION; +#endif +} + +PJ_FIXTURE_EXPORT uint64_t pj_module_create(uint32_t claim_index) { + if (claim_index >= kClaimCount) { + creationError() = "claim index is outside the fixture manifest"; + return PJ_MODULE_CREATION_ERROR_TOKEN; + } + if (claim_index == kCreateFailure) { + creationError() = "fixture creation failure"; + return PJ_MODULE_CREATION_ERROR_TOKEN; + } + const bool claim_is_live = std::any_of( + liveInstances().begin(), liveInstances().end(), + [claim_index](const Instance* instance) { return instance->claim_index == claim_index; }); + if (claim_is_live) { + creationError() = "fixture permits one live instance per claim"; + return PJ_MODULE_CREATION_ERROR_TOKEN; + } + auto* instance = new Instance(claim_index); + liveInstances().insert(instance); + return addressOf(instance); +} + +PJ_FIXTURE_EXPORT void pj_module_destroy(uint64_t token) { + auto* instance = findInstance(token); + if (instance != nullptr) { + liveInstances().erase(instance); + delete instance; + } +} + +PJ_FIXTURE_EXPORT int32_t pj_module_bind(uint64_t token, uint64_t info_addr, uint64_t info_len) { + auto* instance = findInstance(token); + if (instance == nullptr) { + return PJ_MODULE_ERR_BAD_TOKEN; + } + if (info_addr == 0 || info_len > static_cast(SIZE_MAX)) { + return fail(instance, PJ_MODULE_ERR_MALFORMED_INPUT, "binding buffer is unreadable"); + } + const auto* data = reinterpret_cast(static_cast(info_addr)); + auto info = PJ::parser_module::readBindingInfoV1(PJ::Span(data, static_cast(info_len))); + if (!info || info->claim_index != instance->claim_index) { + return fail(instance, PJ_MODULE_ERR_MALFORMED_INPUT, "fixture rejected malformed binding info"); + } + instance->route = info->route; + if (instance->claim_index == kDecline) { + return fail(instance, PJ_MODULE_DECLINE, "fixture bind declined"); + } + instance->error.clear(); + return PJ_MODULE_OK; +} + +PJ_FIXTURE_EXPORT int32_t pj_module_parse( + uint64_t token, uint64_t input_addr, uint64_t input_len, uint64_t output_addr_ptr, uint64_t output_len_ptr) { + auto* instance = findInstance(token); + if (instance == nullptr) { + return PJ_MODULE_ERR_BAD_TOKEN; + } + if (instance->claim_index == kBadToken) { + return fail(instance, PJ_MODULE_ERR_BAD_TOKEN, "fixture forced bad token"); + } + if (input_addr == 0 || output_addr_ptr == 0 || output_len_ptr == 0 || input_len > static_cast(SIZE_MAX)) { + return fail(instance, PJ_MODULE_ERR_MALFORMED_INPUT, "parse buffer is unreadable"); + } + const auto* input_data = reinterpret_cast(static_cast(input_addr)); + auto input = PJ::parser_module::readParseInputV1(PJ::Span(input_data, static_cast(input_len))); + if (!input) { + return fail(instance, PJ_MODULE_ERR_MALFORMED_INPUT, input.error()); + } + + if (instance->claim_index == kDataError) { + return fail(instance, PJ_MODULE_ERR_MALFORMED_INPUT, "fixture payload decode error"); + } + if (instance->claim_index == kMalformed) { + // Deliberately not a decodable output descriptor. + instance->output = {1, 2, 3}; + } else { + // ObjectOutputV1::wire is a borrowed span, so its storage must outlive the + // storeOutput call that serializes it. + std::vector wire; + PJ::parser_module::OutputDescriptorV1 descriptor; + switch (instance->claim_index) { + case kScalar: + descriptor = PJ::parser_module::ScalarOutputV1{ + .has_timestamp = true, + .timestamp_ns = 42, + .fields = + { + {.name = "temperature", .value = 21.5}, + {.name = "status", .value = std::string_view("ready")}, + }, + }; + break; + case kRouteMismatch: + descriptor = PJ::parser_module::ObjectOutputV1{ + .object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .splice = std::nullopt, + .wire = {}, + }; + break; + case kTypeMismatch: + descriptor = PJ::parser_module::ObjectOutputV1{ + .object_type = PJ_BUILTIN_OBJECT_TYPE_IMAGE, + .splice = std::nullopt, + .wire = {}, + }; + break; + case kSplice: + case kSpliceOutOfBounds: + case kSpliceIneligible: + descriptor = PJ::parser_module::ObjectOutputV1{ + .object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .splice = + PJ::parser_module::ObjectSpliceV1{ + .field_number = instance->claim_index == kSpliceIneligible ? uint32_t{8} : uint32_t{9}, + .input_offset = instance->claim_index == kSpliceOutOfBounds + ? static_cast(input->payload.size()) + 1 + : 1, + .input_length = 2, + }, + .wire = {}, + }; + break; + default: { + const std::array point_data{1, 2, 3, 4}; + PJ::sdk::PointCloud cloud; + cloud.width = 1; + cloud.height = 1; + cloud.point_step = 4; + cloud.row_step = 4; + cloud.data = point_data; + wire = PJ::serializePointCloud(cloud); + descriptor = PJ::parser_module::ObjectOutputV1{ + .object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .splice = std::nullopt, + .wire = wire, + }; + break; + } + } + if (const int32_t result = storeOutput(*instance, std::move(descriptor)); result != PJ_MODULE_OK) { + return result; + } + } + + *reinterpret_cast(static_cast(output_addr_ptr)) = addressOf(instance->output.data()); + *reinterpret_cast(static_cast(output_len_ptr)) = instance->output.size(); + return PJ_MODULE_OK; +} + +PJ_FIXTURE_EXPORT uint64_t pj_module_last_error(uint64_t token, uint64_t buffer_addr, uint64_t buffer_cap) { + const Instance* instance = findInstance(token); + const std::string* error = + token == PJ_MODULE_CREATION_ERROR_TOKEN ? &creationError() : (instance == nullptr ? nullptr : &instance->error); + if (error == nullptr || buffer_addr == 0 || buffer_cap == 0) { + return 0; + } + const uint64_t size = std::min(error->size(), buffer_cap); + std::memcpy(reinterpret_cast(static_cast(buffer_addr)), error->data(), static_cast(size)); + return size; +} + +PJ_FIXTURE_EXPORT uint64_t pj_module_alloc(uint64_t size) { + if (size > static_cast(SIZE_MAX)) { + return 0; + } + auto* allocation = new (std::nothrow) uint8_t[static_cast(size)]; + return addressOf(allocation); +} + +#if !defined(PJ_FIXTURE_OMIT_FREE) +PJ_FIXTURE_EXPORT void pj_module_free(uint64_t address, uint64_t) { + delete[] reinterpret_cast(static_cast(address)); +} +#endif + +PJ_FIXTURE_EXPORT uint64_t pj_module_manifest_addr() { +#if defined(PJ_FIXTURE_UNREADABLE_MANIFEST) + return 0; +#else + return addressOf(kManifest); +#endif +} + +PJ_FIXTURE_EXPORT uint64_t pj_module_manifest_len() { + return sizeof(kManifest) - 1; +} + +} // extern "C" diff --git a/pj_plugins/tests/native_parser_module_fixture.hpp b/pj_plugins/tests/native_parser_module_fixture.hpp new file mode 100644 index 00000000..e12d168a --- /dev/null +++ b/pj_plugins/tests/native_parser_module_fixture.hpp @@ -0,0 +1,28 @@ +#pragma once +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +/// Claim indices exposed by native_parser_module_fixture.cpp. The order must +/// match the claims array of the fixture manifest: the host addresses claims +/// positionally, so a reordering here silently rebinds every test. +namespace pj_fixture { + +enum ClaimIndex : uint32_t { + kObject = 0, + kScalar = 1, + kDecline = 2, + kCreateFailure = 3, + kDataError = 4, + kMalformed = 5, + kSplice = 6, + kSpliceOutOfBounds = 7, + kSpliceIneligible = 8, + kBadToken = 9, + kRouteMismatch = 10, + kTypeMismatch = 11, + kClaimCount = 12, +}; + +} // namespace pj_fixture diff --git a/pj_plugins/tests/native_parser_module_test.cpp b/pj_plugins/tests/native_parser_module_test.cpp new file mode 100644 index 00000000..ad8bdb92 --- /dev/null +++ b/pj_plugins/tests/native_parser_module_test.cpp @@ -0,0 +1,70 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/native_parser_module.hpp" + +#include + +#include +#include + +#include "native_parser_module_fixture.hpp" +#include "pj_base/parser_module_abi.h" +#include "pj_plugins/host/parser_claim_catalog.hpp" + +namespace PJ { +namespace { + +TEST(NativeParserModule, LoadsCompleteAbiAndCopiesManifestForCatalogAdmission) { + std::vector diagnostics; + auto module = NativeParserModule::load( + PJ_NATIVE_MODULE_FIXTURE_PATH, [&](const Diagnostic& diagnostic) { diagnostics.push_back(diagnostic); }); + + ASSERT_TRUE(module.has_value()) << module.error(); + EXPECT_TRUE(module->valid()); + EXPECT_EQ(module->path(), PJ_NATIVE_MODULE_FIXTURE_PATH); + EXPECT_NE(module->manifestJson().find("org.plotjuggler.test.native-module"), std::string_view::npos); + EXPECT_TRUE(diagnostics.empty()); + + ParserClaimCatalog catalog; + auto manifest = catalog.ingestModuleManifest(module->manifestJson(), ParserClaimProvenance::kFolderDrop, 7); + ASSERT_TRUE(manifest.has_value()) << manifest.error(); + EXPECT_EQ(manifest->id, "org.plotjuggler.test.native-module"); + EXPECT_EQ(manifest->claims.size(), pj_fixture::kClaimCount); + EXPECT_EQ(catalog.claims().size(), pj_fixture::kClaimCount); +} + +TEST(NativeParserModule, RejectsEachLoaderFailureWithOneDiagnostic) { + for (const std::string path : { + PJ_NATIVE_MODULE_MISSING_EXPORT_PATH, + PJ_NATIVE_MODULE_WRONG_ABI_PATH, + PJ_NATIVE_MODULE_UNREADABLE_MANIFEST_PATH, + }) { + std::vector diagnostics; + auto module = + NativeParserModule::load(path, [&](const Diagnostic& diagnostic) { diagnostics.push_back(diagnostic); }); + + EXPECT_FALSE(module.has_value()) << path; + ASSERT_EQ(diagnostics.size(), 1U) << path; + EXPECT_EQ(diagnostics.front().level, DiagnosticLevel::kError); + EXPECT_EQ(diagnostics.front().id, path); + EXPECT_EQ(diagnostics.front().message, module.error()); + } +} + +TEST(NativeParserModule, ReportsSpecificLoaderFailureCauses) { + auto missing = NativeParserModule::load(PJ_NATIVE_MODULE_MISSING_EXPORT_PATH); + ASSERT_FALSE(missing.has_value()); + EXPECT_NE(missing.error().find(PJ_MODULE_FREE_EXPORT_NAME), std::string::npos); + + auto wrong_abi = NativeParserModule::load(PJ_NATIVE_MODULE_WRONG_ABI_PATH); + ASSERT_FALSE(wrong_abi.has_value()); + EXPECT_NE(wrong_abi.error().find("ABI mismatch"), std::string::npos); + + auto unreadable = NativeParserModule::load(PJ_NATIVE_MODULE_UNREADABLE_MANIFEST_PATH); + ASSERT_FALSE(unreadable.has_value()); + EXPECT_NE(unreadable.error().find("manifest is unreadable"), std::string::npos); +} + +} // namespace +} // namespace PJ diff --git a/pj_plugins/tests/parser_claim_catalog_test.cpp b/pj_plugins/tests/parser_claim_catalog_test.cpp new file mode 100644 index 00000000..18ca47c7 --- /dev/null +++ b/pj_plugins/tests/parser_claim_catalog_test.cpp @@ -0,0 +1,334 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_claim_catalog.hpp" + +#include + +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_module_abi.h" +#include "pj_base/parser_route_claims_protocol.h" + +namespace PJ { +namespace { + +DiagnosticSink recordInto(std::vector& diagnostics) { + return [&diagnostics](const Diagnostic& diagnostic) { diagnostics.push_back(diagnostic); }; +} + +ParserClaim scalarClaim(std::string provider = "provider", std::string claim_id = "claim") { + return ParserClaim{ + .encoding = "protobuf", + .type_name = "foxglove.PointCloud", + .route_flags = PJ_PARSER_ROUTE_FLAG_SCALAR_V1, + .object_type = std::nullopt, + .schema_digests = {}, + .provider_id = std::move(provider), + .claim_id = std::move(claim_id), + .priority = 0, + .provenance = ParserClaimProvenance::kFolderDrop, + }; +} + +ParserClaim objectClaim(std::string type_name = "sensor_msgs/msg/PointCloud2") { + return ParserClaim{ + .encoding = "ros2msg", + .type_name = std::move(type_name), + .route_flags = PJ_PARSER_ROUTE_FLAG_OBJECT_V1, + .object_type = sdk::BuiltinObjectType::kPointCloud, + .schema_digests = {}, + .provider_id = "provider", + .claim_id = "object", + .priority = 0, + .provenance = ParserClaimProvenance::kFolderDrop, + }; +} + +void expectAdmissionRejected(ParserClaim claim, std::string_view diagnostic_fragment) { + std::vector diagnostics; + ParserClaimCatalog catalog(recordInto(diagnostics)); + + const auto result = catalog.admitClaims({std::move(claim)}, ParserClaimProvenance::kMarketplace, 17); + + ASSERT_FALSE(result.has_value()); + EXPECT_NE(result.error().find(diagnostic_fragment), std::string::npos) << result.error(); + ASSERT_EQ(diagnostics.size(), 1U); + EXPECT_EQ(diagnostics.front().level, DiagnosticLevel::kError); + EXPECT_NE(diagnostics.front().message.find(diagnostic_fragment), std::string::npos); + EXPECT_TRUE(catalog.claims().empty()); +} + +TEST(ParserClaimCatalog, AdmissionRejectsEveryBoundedPriorityViolation) { + for (const int32_t priority : {-1001, 1001}) { + SCOPED_TRACE(priority); + auto claim = scalarClaim(); + claim.priority = priority; + expectAdmissionRejected(std::move(claim), "[-1000,1000]"); + } + + ParserClaimCatalog catalog; + auto low = scalarClaim("provider", "low"); + low.priority = -1000; + auto high = scalarClaim("provider", "high"); + high.priority = 1000; + EXPECT_TRUE(catalog.admitClaims({low, high}, ParserClaimProvenance::kBundled, 1).has_value()); +} + +TEST(ParserClaimCatalog, AdmissionRejectsWildcardObjectClaims) { + auto claim = objectClaim("*"); + expectAdmissionRejected(std::move(claim), "wildcard"); +} + +TEST(ParserClaimCatalog, AdmissionRejectsObjectTypeWithoutObjectRoute) { + auto claim = scalarClaim(); + claim.object_type = sdk::BuiltinObjectType::kImage; + expectAdmissionRejected(std::move(claim), "forbidden"); +} + +TEST(ParserClaimCatalog, AdmissionRejectsObjectRouteWithoutObjectType) { + auto claim = objectClaim(); + claim.object_type.reset(); + expectAdmissionRejected(std::move(claim), "requires object_type"); +} + +TEST(ParserClaimCatalog, AdmissionRejectsUnknownEncodingCaseSensitively) { + for (const std::string_view encoding : {"unknown", "ROS2MSG"}) { + SCOPED_TRACE(encoding); + auto claim = scalarClaim(); + claim.encoding = encoding; + expectAdmissionRejected(std::move(claim), "unknown encoding"); + } +} + +TEST(ParserClaimCatalog, AdmissionRejectsUnknownObjectType) { + auto claim = objectClaim(); + claim.object_type = static_cast(999); + expectAdmissionRejected(std::move(claim), "unknown object_type"); +} + +TEST(ParserClaimCatalog, AdmissionRejectsMalformedSchemaDigestAllowLists) { + for (const std::string& digest : { + std::string{}, + std::string("sha256:short"), + "sha256:" + std::string(63, 'a'), + "sha256:" + std::string(63, 'a') + "g", + "SHA256:" + std::string(64, 'a'), + }) { + SCOPED_TRACE(digest); + auto claim = scalarClaim(); + claim.schema_digests = {digest}; + expectAdmissionRejected(std::move(claim), "sha256:<64-hex>"); + } +} + +TEST(ParserClaimCatalog, AdmissionRejectsDuplicateIdentityTransactionally) { + std::vector diagnostics; + ParserClaimCatalog catalog(recordInto(diagnostics)); + ASSERT_TRUE(catalog.admitClaims({scalarClaim("same", "one")}, ParserClaimProvenance::kFolderDrop, 1).has_value()); + + auto duplicate = scalarClaim("same", "one"); + auto otherwise_valid = scalarClaim("same", "two"); + const auto result = + catalog.admitClaims({std::move(otherwise_valid), std::move(duplicate)}, ParserClaimProvenance::kBundled, 2); + + ASSERT_FALSE(result.has_value()); + EXPECT_NE(result.error().find("duplicate"), std::string::npos); + ASSERT_EQ(catalog.claims().size(), 1U); + EXPECT_EQ(catalog.claims().front().claim.claim_id, "one"); + ASSERT_EQ(diagnostics.size(), 1U); +} + +TEST(ParserClaimCatalog, AdmissionNormalizesNamesAndUsesHostProvenance) { + ParserClaimCatalog catalog; + auto claim = objectClaim("sensor_msgs/PointCloud2"); + claim.provenance = ParserClaimProvenance::kFolderDrop; + + ASSERT_TRUE(catalog.admitClaims({claim}, ParserClaimProvenance::kBundled, 42).has_value()); + ASSERT_EQ(catalog.claims().size(), 1U); + EXPECT_EQ(catalog.claims()[0].claim.type_name, "sensor_msgs/msg/PointCloud2"); + EXPECT_EQ(catalog.claims()[0].claim.provenance, ParserClaimProvenance::kBundled); + EXPECT_EQ(catalog.claims()[0].provider_generation, 42U); +} + +TEST(ParserClaimCatalog, ModuleManifestDecodesValidatedClaimsAndMetadata) { + const auto manifest = decodeParserModuleManifest( + R"({ + "module_abi": 1, + "id": "com.example.radar", + "name": "Radar Parsers", + "version": "1.2.3-rc.1+build.7", + "claims": [ + {"claim_id":"object","encoding":"ros2msg","type_name":"radar_msgs/RadarScan", + "routes":["scalar","object"],"object_type":"kPointCloud", + "schema_digests":[ + "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"],"priority":17}, + {"claim_id":"fallback","encoding":"protobuf","type_name":".example.RadarScan", + "routes":["scalar"],"priority":-2} + ] + })", + ParserClaimProvenance::kMarketplace); + + ASSERT_TRUE(manifest.has_value()) << manifest.error(); + EXPECT_EQ(manifest->id, "com.example.radar"); + EXPECT_EQ(manifest->name, "Radar Parsers"); + EXPECT_EQ(manifest->version, "1.2.3-rc.1+build.7"); + ASSERT_EQ(manifest->claims.size(), 2U); + EXPECT_EQ(manifest->claims[0].type_name, "radar_msgs/msg/RadarScan"); + EXPECT_EQ(manifest->claims[0].route_flags, PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1); + EXPECT_EQ(manifest->claims[0].object_type, sdk::BuiltinObjectType::kPointCloud); + EXPECT_EQ(manifest->claims[0].schema_digests.size(), 2U); + EXPECT_EQ(manifest->claims[0].provenance, ParserClaimProvenance::kMarketplace); + EXPECT_EQ(manifest->claims[1].type_name, "example.RadarScan"); + EXPECT_FALSE(manifest->claims[1].object_type.has_value()); +} + +TEST(ParserClaimCatalog, ModuleManifestRejectsMalformedJsonAndAbiVersion) { + for (const std::string_view manifest : { + "not json", + R"([])", + R"({"module_abi":2,"id":"x","name":"X","version":"1.0.0","claims":[]})", + R"({"id":"x","name":"X","version":"1.0.0","claims":[]})", + }) { + SCOPED_TRACE(manifest); + const auto parsed = decodeParserModuleManifest(manifest, ParserClaimProvenance::kFolderDrop); + EXPECT_FALSE(parsed.has_value()); + } +} + +TEST(ParserClaimCatalog, MalformedModuleManifestEmitsOneDiagnosticAndAdmitsNothing) { + std::vector diagnostics; + ParserClaimCatalog catalog(recordInto(diagnostics)); + + const auto result = catalog.ingestModuleManifest("not json", ParserClaimProvenance::kFolderDrop, 1); + + ASSERT_FALSE(result.has_value()); + ASSERT_EQ(diagnostics.size(), 1U); + EXPECT_EQ(diagnostics.front().level, DiagnosticLevel::kError); + EXPECT_NE(diagnostics.front().message.find("invalid JSON"), std::string::npos); + EXPECT_TRUE(catalog.claims().empty()); +} + +TEST(ParserClaimCatalog, ModuleManifestRequiresStableIdentityNameAndThreePartSemver) { + for (const std::string_view manifest : { + R"({"module_abi":1,"name":"X","version":"1.0.0","claims":[]})", + R"({"module_abi":1,"id":"x","version":"1.0.0","claims":[]})", + R"({"module_abi":1,"id":"x","name":"X","version":"1.0","claims":[]})", + }) { + SCOPED_TRACE(manifest); + const auto parsed = decodeParserModuleManifest(manifest, ParserClaimProvenance::kBundled); + EXPECT_FALSE(parsed.has_value()); + } +} + +TEST(ParserClaimCatalog, ModuleManifestRejectsDuplicateClaimsAsAWhole) { + std::vector diagnostics; + ParserClaimCatalog catalog(recordInto(diagnostics)); + const auto result = catalog.ingestModuleManifest( + R"({"module_abi":1,"id":"module","name":"Module","version":"1.0.0","claims":[ + {"claim_id":"same","encoding":"protobuf","type_name":"a.Type","routes":["scalar"],"priority":0}, + {"claim_id":"same","encoding":"protobuf","type_name":"b.Type","routes":["scalar"],"priority":0} + ]})", + ParserClaimProvenance::kFolderDrop, 1); + + ASSERT_FALSE(result.has_value()); + EXPECT_NE(result.error().find("duplicate"), std::string::npos); + EXPECT_TRUE(catalog.claims().empty()); + ASSERT_EQ(diagnostics.size(), 1U); +} + +TEST(ParserClaimCatalog, ModuleManifestCannotForgeProvenance) { + const auto result = decodeParserModuleManifest( + R"({"module_abi":1,"id":"module","name":"Module","version":"1.0.0", + "provenance":"bundled","claims":[]})", + ParserClaimProvenance::kFolderDrop); + + ASSERT_FALSE(result.has_value()); + EXPECT_NE(result.error().find("provenance"), std::string::npos); +} + +TEST(ParserClaimCatalog, ParserPluginSynthesisFreezesWildcardAndHandlerIds) { + const std::vector encodings{"ros2msg", "protobuf"}; + const std::vector exact{ + { + .encoding = "ros2msg", + .type_name = "sensor_msgs/PointCloud2", + .classification = + { + .route_flags = PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1, + .match = PJ_PARSER_ROUTE_MATCH_EXACT_V1, + .status = PJ_PARSER_ROUTE_STATUS_CLAIMED_V1, + .object_type = static_cast(sdk::BuiltinObjectType::kPointCloud), + }, + .schema_digests = {}, + }, + { + .encoding = "protobuf", + .type_name = "foxglove.RawImage", + .classification = + { + .route_flags = 0, + .match = PJ_PARSER_ROUTE_MATCH_EXACT_V1, + .status = PJ_PARSER_ROUTE_STATUS_DECLINED_V1, + .object_type = static_cast(sdk::BuiltinObjectType::kNone), + }, + .schema_digests = {}, + }, + }; + + const auto claims = synthesizeParserPluginClaims("parser", encodings, exact, ParserClaimProvenance::kBundled); + + ASSERT_TRUE(claims.has_value()) << claims.error(); + ASSERT_EQ(claims->size(), 3U); + EXPECT_EQ((*claims)[0].claim_id, "wildcard:ros2msg"); + EXPECT_EQ((*claims)[1].claim_id, "wildcard:protobuf"); + EXPECT_EQ((*claims)[2].claim_id, "handler:ros2msg:sensor_msgs/msg/PointCloud2"); + EXPECT_EQ((*claims)[2].type_name, "sensor_msgs/msg/PointCloud2"); +} + +TEST(ParserClaimCatalog, ParserPluginSynthesisRejectsUnknownEncodingsAndMalformedClassifications) { + const std::vector unknown{"ROS2MSG"}; + EXPECT_FALSE(synthesizeParserPluginClaims("parser", unknown, {}, ParserClaimProvenance::kBundled).has_value()); + + const std::vector encodings{"protobuf"}; + const std::vector malformed{{ + .encoding = "protobuf", + .type_name = "foxglove.RawImage", + .classification = + { + .route_flags = PJ_PARSER_ROUTE_FLAG_SCALAR_V1, + .match = 1, + .status = PJ_PARSER_ROUTE_STATUS_CLAIMED_V1, + .object_type = static_cast(sdk::BuiltinObjectType::kNone), + }, + .schema_digests = {}, + }}; + const auto result = synthesizeParserPluginClaims("parser", encodings, malformed, ParserClaimProvenance::kBundled); + ASSERT_FALSE(result.has_value()); + EXPECT_NE(result.error().find("non-exact"), std::string::npos); +} + +TEST(ParserClaimCatalog, CatalogMutationGenerationChangesOnlyWhenStateChanges) { + ParserClaimCatalog catalog; + EXPECT_EQ(catalog.generation(), 0U); + ASSERT_TRUE(catalog.admitClaims({}, ParserClaimProvenance::kBundled, 1).has_value()); + EXPECT_EQ(catalog.generation(), 0U); + ASSERT_TRUE(catalog.admitClaims({scalarClaim()}, ParserClaimProvenance::kBundled, 1).has_value()); + EXPECT_EQ(catalog.generation(), 1U); + EXPECT_FALSE(catalog.removeProvider("missing")); + EXPECT_EQ(catalog.generation(), 1U); + EXPECT_TRUE(catalog.removeProvider("provider")); + EXPECT_EQ(catalog.generation(), 2U); + catalog.clear(); + EXPECT_EQ(catalog.generation(), 2U); +} + +} // namespace +} // namespace PJ diff --git a/pj_plugins/tests/parser_module_authoring_e2e_test.cpp b/pj_plugins/tests/parser_module_authoring_e2e_test.cpp new file mode 100644 index 00000000..bba6e817 --- /dev/null +++ b/pj_plugins/tests/parser_module_authoring_e2e_test.cpp @@ -0,0 +1,129 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include + +#include +#include +#include +#include +#include +#include + +#include "pj_base/builtin/point_cloud.hpp" +#include "pj_base/builtin_object_abi.h" +#include "pj_base/parser_module_abi.h" +#include "pj_plugins/host/native_parser_module.hpp" +#include "pj_plugins/host/parser_claim_catalog.hpp" +#include "pj_plugins/host/parser_module_runtime.hpp" + +namespace PJ { +namespace { + +constexpr std::string_view kSchema = "uint32 width\nstring frame_id\nuint8[] data\n"; + +Span bytes(std::string_view text) { + return {reinterpret_cast(text.data()), text.size()}; +} + +void appendU32(std::vector& output, uint32_t value) { + const size_t relative = output.size() - 4; + output.insert(output.end(), (4 - (relative % 4)) % 4, 0); + for (size_t index = 0; index < 4; ++index) { + output.push_back(static_cast(value >> (index * 8U))); + } +} + +std::vector toyPayload() { + std::vector output{0, 1, 0, 0}; + appendU32(output, 2); + appendU32(output, 4); + output.insert(output.end(), {'m', 'a', 'p', 0}); + appendU32(output, 8); + output.insert(output.end(), {1, 2, 3, 4, 5, 6, 7, 8}); + return output; +} + +parser_module::BindingInfoV1 binding(uint32_t claim_index, std::string_view schema) { + return parser_module::BindingInfoV1{ + .route = parser_module::Route::kObject, + .claim_index = claim_index, + .expected_object_type = PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD, + .encoding = bytes("ros2msg"), + .type_name = bytes(claim_index == 0 ? "toy_msgs/msg/Cloud" : "toy_msgs/msg/CloudSplice"), + .schema = bytes(schema), + .claim_id = bytes(claim_index == 0 ? "full-wire" : "spliced"), + .config_json = bytes("{}"), + .schema_digest = {}, + }; +} + +TEST(ParserModuleAuthoringE2E, LoadsAdmitsBindsAndParsesFullAndSplicedPointCloud) { + auto module = NativeParserModule::load(PJ_TOY_CDR_POINTCLOUD_MODULE_PATH); + ASSERT_TRUE(module.has_value()) << module.error(); + ParserClaimCatalog catalog; + auto manifest = catalog.ingestModuleManifest(module->manifestJson(), ParserClaimProvenance::kFolderDrop, 12); + ASSERT_TRUE(manifest.has_value()) << manifest.error(); + ASSERT_EQ(manifest->claims.size(), 2U); + + const auto payload = toyPayload(); + const parser_module::ParseInputV1 input{ + .has_timestamp = true, + .timestamp_ns = 4242, + .payload = payload, + }; + + auto full = NativeParserModuleInstance::create(*module, 0); + ASSERT_TRUE(full.has_value()) << full.error(); + auto full_bind = full->bind(binding(0, kSchema)); + ASSERT_TRUE(full_bind.has_value()) << full_bind.error(); + ASSERT_EQ(full_bind->outcome, ParserModuleBindOutcome::kAccept); + auto full_result = full->parse(input); + ASSERT_TRUE(full_result.has_value()) << full_result.error(); + ASSERT_EQ(full_result->fault, ParserModuleFaultKind::kNone) << full_result->message; + const auto* full_object = std::get_if(&*full_result->output); + ASSERT_NE(full_object, nullptr); + EXPECT_FALSE(full_object->splice.has_value()); + const auto* full_cloud = std::any_cast(&full_object->object); + ASSERT_NE(full_cloud, nullptr); + EXPECT_EQ(full_cloud->width, 2U); + EXPECT_EQ(full_cloud->frame_id, "map"); + EXPECT_EQ(full_cloud->data.size(), 8U); + EXPECT_EQ(full_cloud->data[7], 8U); + EXPECT_EQ(full_cloud->timestamp_ns, 4242); + + auto spliced = NativeParserModuleInstance::create(*module, 1); + ASSERT_TRUE(spliced.has_value()) << spliced.error(); + auto splice_bind = spliced->bind(binding(1, kSchema)); + ASSERT_TRUE(splice_bind.has_value()) << splice_bind.error(); + ASSERT_EQ(splice_bind->outcome, ParserModuleBindOutcome::kAccept); + auto splice_result = spliced->parse(input); + ASSERT_TRUE(splice_result.has_value()) << splice_result.error(); + ASSERT_EQ(splice_result->fault, ParserModuleFaultKind::kNone) << splice_result->message; + const auto* splice_object = std::get_if(&*splice_result->output); + ASSERT_NE(splice_object, nullptr); + ASSERT_TRUE(splice_object->splice.has_value()); + EXPECT_EQ(splice_object->splice->field_number, 9U); + EXPECT_EQ(splice_object->splice->input_offset, 20U); + EXPECT_EQ(splice_object->splice->payload_bytes, (std::vector{1, 2, 3, 4, 5, 6, 7, 8})); + const auto* splice_cloud = std::any_cast(&splice_object->object); + ASSERT_NE(splice_cloud, nullptr); + EXPECT_EQ(splice_cloud->timestamp_ns, 4242); + ASSERT_EQ(splice_cloud->data.size(), 8U); + EXPECT_EQ(splice_cloud->data[0], 1U); + EXPECT_EQ(splice_cloud->data[7], 8U); +} + +TEST(ParserModuleAuthoringE2E, DeclinesUnsupportedSchemaRevisionAtBind) { + auto module = NativeParserModule::load(PJ_TOY_CDR_POINTCLOUD_MODULE_PATH); + ASSERT_TRUE(module.has_value()) << module.error(); + auto instance = NativeParserModuleInstance::create(*module, 0); + ASSERT_TRUE(instance.has_value()) << instance.error(); + auto result = instance->bind(binding(0, "uint32 width\nstring frame_id\n")); + ASSERT_TRUE(result.has_value()) << result.error(); + EXPECT_EQ(result->outcome, ParserModuleBindOutcome::kDecline); + EXPECT_NE(result->message.find("unsupported toy schema revision"), std::string::npos); +} + +} // namespace +} // namespace PJ diff --git a/pj_plugins/tests/parser_module_runtime_test.cpp b/pj_plugins/tests/parser_module_runtime_test.cpp new file mode 100644 index 00000000..2ec65341 --- /dev/null +++ b/pj_plugins/tests/parser_module_runtime_test.cpp @@ -0,0 +1,218 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_module_runtime.hpp" + +#include + +#include +#include +#include +#include +#include + +#include "native_parser_module_fixture.hpp" +#include "pj_base/builtin/point_cloud.hpp" +#include "pj_base/builtin_object_abi.h" +#include "pj_base/span.hpp" + +namespace PJ { +namespace { + +using enum pj_fixture::ClaimIndex; + +Span bytes(std::string_view text) { + return {reinterpret_cast(text.data()), text.size()}; +} + +parser_module::BindingInfoV1 binding( + uint32_t claim_index, parser_module::Route route, uint16_t object_type = PJ_BUILTIN_OBJECT_TYPE_NONE) { + return parser_module::BindingInfoV1{ + .route = route, + .claim_index = claim_index, + .expected_object_type = object_type, + .encoding = bytes("protobuf"), + .type_name = bytes("fixture.Message"), + .schema = {}, + .claim_id = bytes("fixture-claim"), + .config_json = bytes("{}"), + .schema_digest = bytes("sha256:test"), + }; +} + +parser_module::ParseInputV1 input() { + static constexpr std::array kPayload{10, 20, 30, 40}; + return parser_module::ParseInputV1{ + .has_timestamp = true, + .timestamp_ns = 100, + .payload = kPayload, + }; +} + +NativeParserModule loadFixture() { + auto module = NativeParserModule::load(PJ_NATIVE_MODULE_FIXTURE_PATH); + EXPECT_TRUE(module.has_value()) << module.error(); + return module ? std::move(*module) : NativeParserModule{}; +} + +NativeParserModuleInstance createBound( + const NativeParserModule& module, uint32_t claim_index, parser_module::Route route, + uint16_t object_type = PJ_BUILTIN_OBJECT_TYPE_NONE) { + auto instance = NativeParserModuleInstance::create(module, claim_index); + EXPECT_TRUE(instance.has_value()) << instance.error(); + if (!instance) { + return {}; + } + auto result = instance->bind(binding(claim_index, route, object_type)); + EXPECT_TRUE(result.has_value()) << result.error(); + if (result) { + EXPECT_EQ(result->outcome, ParserModuleBindOutcome::kAccept); + EXPECT_EQ(result->fault, ParserModuleFaultKind::kNone); + } + return std::move(*instance); +} + +TEST(ParserModuleRuntime, RunsObjectAndScalarLifecyclesAndOwnsOutputs) { + auto module = loadFixture(); + ASSERT_TRUE(module.valid()); + + auto object_instance = createBound(module, kObject, parser_module::Route::kObject, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + auto object_result = object_instance.parse(input()); + ASSERT_TRUE(object_result.has_value()) << object_result.error(); + ASSERT_EQ(object_result->fault, ParserModuleFaultKind::kNone); + ASSERT_TRUE(object_result->output.has_value()); + const auto* object = std::get_if(&*object_result->output); + ASSERT_NE(object, nullptr); + EXPECT_EQ(sdk::typeOf(object->object), sdk::BuiltinObjectType::kPointCloud); + EXPECT_FALSE(object->wire.empty()); + EXPECT_FALSE(object->splice.has_value()); + const auto* cloud = std::any_cast(&object->object); + ASSERT_NE(cloud, nullptr); + EXPECT_EQ(cloud->data.size(), 4U); + EXPECT_EQ(cloud->data[2], 3U); + + auto scalar_instance = createBound(module, kScalar, parser_module::Route::kScalar); + auto scalar_result = scalar_instance.parse(input()); + ASSERT_TRUE(scalar_result.has_value()) << scalar_result.error(); + ASSERT_EQ(scalar_result->fault, ParserModuleFaultKind::kNone); + ASSERT_TRUE(scalar_result->output.has_value()); + const auto* scalar = std::get_if(&*scalar_result->output); + ASSERT_NE(scalar, nullptr); + EXPECT_TRUE(scalar->has_timestamp); + EXPECT_EQ(scalar->timestamp_ns, 42); + ASSERT_EQ(scalar->fields.size(), 2U); + EXPECT_EQ(scalar->fields[0].name, "temperature"); + EXPECT_DOUBLE_EQ(std::get(scalar->fields[0].value), 21.5); + EXPECT_EQ(std::get(scalar->fields[1].value), "ready"); +} + +TEST(ParserModuleRuntime, SurfacesBindDeclineAndTokenZeroCreationError) { + auto module = loadFixture(); + + auto declined = NativeParserModuleInstance::create(module, kDecline); + ASSERT_TRUE(declined.has_value()) << declined.error(); + auto bind_result = declined->bind(binding(kDecline, parser_module::Route::kScalar)); + ASSERT_TRUE(bind_result.has_value()) << bind_result.error(); + EXPECT_EQ(bind_result->outcome, ParserModuleBindOutcome::kDecline); + EXPECT_EQ(bind_result->fault, ParserModuleFaultKind::kNone); + EXPECT_EQ(bind_result->message, "fixture bind declined"); + EXPECT_FALSE(declined->parse(input()).has_value()); + + auto failed = NativeParserModuleInstance::create(module, kCreateFailure); + ASSERT_FALSE(failed.has_value()); + EXPECT_EQ(failed.error(), "fixture creation failure"); +} + +TEST(ParserModuleRuntime, ClassifiesDataErrorsSeparatelyFromContractViolations) { + auto module = loadFixture(); + + auto data_instance = createBound(module, kDataError, parser_module::Route::kScalar); + auto data_result = data_instance.parse(input()); + ASSERT_TRUE(data_result.has_value()) << data_result.error(); + EXPECT_EQ(data_result->fault, ParserModuleFaultKind::kDataError); + EXPECT_EQ(data_result->result_code, PJ_MODULE_ERR_MALFORMED_INPUT); + EXPECT_EQ(data_result->message, "fixture payload decode error"); + + for (const uint32_t claim_index : {kMalformed, kBadToken, kRouteMismatch}) { + auto instance = createBound(module, claim_index, parser_module::Route::kScalar); + auto result = instance.parse(input()); + ASSERT_TRUE(result.has_value()) << result.error(); + EXPECT_EQ(result->fault, ParserModuleFaultKind::kContractViolation) << claim_index; + } + + auto mismatch = createBound(module, kTypeMismatch, parser_module::Route::kObject, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + auto mismatch_result = mismatch.parse(input()); + ASSERT_TRUE(mismatch_result.has_value()) << mismatch_result.error(); + EXPECT_EQ(mismatch_result->fault, ParserModuleFaultKind::kContractViolation); + EXPECT_NE(mismatch_result->message.find("does not match"), std::string::npos); +} + +TEST(ParserModuleRuntime, AcceptsEligibleSpliceAndRejectsInvalidReferences) { + auto module = loadFixture(); + + auto valid = createBound(module, kSplice, parser_module::Route::kObject, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + auto valid_result = valid.parse(input()); + ASSERT_TRUE(valid_result.has_value()) << valid_result.error(); + ASSERT_EQ(valid_result->fault, ParserModuleFaultKind::kNone); + const auto* valid_object = std::get_if(&*valid_result->output); + ASSERT_NE(valid_object, nullptr); + ASSERT_TRUE(valid_object->splice.has_value()); + EXPECT_EQ(valid_object->splice->field_number, 9U); + EXPECT_EQ(valid_object->splice->input_offset, 1U); + EXPECT_EQ(valid_object->splice->payload_bytes, (std::vector{20, 30})); + const auto* valid_cloud = std::any_cast(&valid_object->object); + ASSERT_NE(valid_cloud, nullptr); + ASSERT_EQ(valid_cloud->data.size(), 2U); + EXPECT_EQ(valid_cloud->data[0], 20U); + EXPECT_EQ(valid_cloud->data[1], 30U); + + auto out_of_bounds = + createBound(module, kSpliceOutOfBounds, parser_module::Route::kObject, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + auto out_of_bounds_result = out_of_bounds.parse(input()); + ASSERT_TRUE(out_of_bounds_result.has_value()) << out_of_bounds_result.error(); + EXPECT_EQ(out_of_bounds_result->fault, ParserModuleFaultKind::kContractViolation); + EXPECT_NE(out_of_bounds_result->message.find("outside"), std::string::npos); + + auto ineligible = + createBound(module, kSpliceIneligible, parser_module::Route::kObject, PJ_BUILTIN_OBJECT_TYPE_POINTCLOUD); + auto ineligible_result = ineligible.parse(input()); + ASSERT_TRUE(ineligible_result.has_value()) << ineligible_result.error(); + EXPECT_EQ(ineligible_result->fault, ParserModuleFaultKind::kContractViolation); + EXPECT_NE(ineligible_result->message.find("not eligible"), std::string::npos); +} + +TEST(ParserModuleRuntime, StrikeTrackerQuarantinesReplaysAndThenDisables) { + auto module = loadFixture(); + const ParserModuleClaimKey key{"org.plotjuggler.test.native-module", "malformed"}; + ParserModuleStrikeTracker tracker; + + auto first = createBound(module, kMalformed, parser_module::Route::kScalar); + for (int strike = 1; strike <= 3; ++strike) { + auto result = first.parse(input()); + ASSERT_TRUE(result.has_value()) << result.error(); + const auto state = tracker.recordFault(key, result->fault); + EXPECT_EQ(state.health, strike == 3 ? ParserModuleClaimHealth::kQuarantined : ParserModuleClaimHealth::kActive); + } + EXPECT_EQ(tracker.state(key).quarantine_count, 1U); + + first = {}; + auto replay = createBound(module, kMalformed, parser_module::Route::kScalar); + ASSERT_TRUE(replay.valid()); + ASSERT_TRUE(tracker.markRecreated(key)); + EXPECT_EQ(tracker.state(key).health, ParserModuleClaimHealth::kActive); + + EXPECT_EQ(tracker.recordFault(key, ParserModuleFaultKind::kDataError).health, ParserModuleClaimHealth::kActive); + EXPECT_EQ(tracker.state(key).strikes, 0U); + for (int strike = 1; strike <= 3; ++strike) { + auto result = replay.parse(input()); + ASSERT_TRUE(result.has_value()) << result.error(); + const auto state = tracker.recordFault(key, result->fault); + EXPECT_EQ(state.health, strike == 3 ? ParserModuleClaimHealth::kDisabled : ParserModuleClaimHealth::kActive); + } + EXPECT_FALSE(tracker.markRecreated(key)); + EXPECT_EQ(tracker.state(key).health, ParserModuleClaimHealth::kDisabled); + EXPECT_EQ(tracker.state(key).quarantine_count, 2U); +} + +} // namespace +} // namespace PJ diff --git a/pj_plugins/tests/parser_route_resolver_test.cpp b/pj_plugins/tests/parser_route_resolver_test.cpp new file mode 100644 index 00000000..ffe2536d --- /dev/null +++ b/pj_plugins/tests/parser_route_resolver_test.cpp @@ -0,0 +1,382 @@ +// Copyright 2026 Davide Faconti +// SPDX-License-Identifier: Apache-2.0 + +#include "pj_plugins/host/parser_route_resolver.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pj_base/parser_route_claims_protocol.h" + +namespace PJ { +namespace { + +DiagnosticSink recordInto(std::vector& diagnostics) { + return [&diagnostics](const Diagnostic& diagnostic) { diagnostics.push_back(diagnostic); }; +} + +ParserClaim claim( + std::string provider, std::string claim_id, std::string type_name = "foxglove.PointCloud", + uint16_t routes = PJ_PARSER_ROUTE_FLAG_SCALAR_V1, int32_t priority = 0, + ParserClaimProvenance provenance = ParserClaimProvenance::kFolderDrop) { + ParserClaim result{ + .encoding = "protobuf", + .type_name = std::move(type_name), + .route_flags = routes, + .object_type = std::nullopt, + .schema_digests = {}, + .provider_id = std::move(provider), + .claim_id = std::move(claim_id), + .priority = priority, + .provenance = provenance, + }; + if ((routes & PJ_PARSER_ROUTE_FLAG_OBJECT_V1) != 0) { + result.object_type = sdk::BuiltinObjectType::kPointCloud; + } + return result; +} + +ParserProbeDecision probeDecision( + ParserProbeOutcome outcome, std::string diagnostic = {}, std::shared_ptr retained_instance = {}) { + return { + .outcome = outcome, + .diagnostic = std::move(diagnostic), + .retained_instance = std::move(retained_instance), + }; +} + +std::string schemaDigest(char digit) { + return "sha256:" + std::string(64, digit); +} + +void admit(ParserClaimCatalog& catalog, std::vector claims, uint64_t generation = 1) { + const ParserClaimProvenance provenance = claims.front().provenance; + const auto result = catalog.admitClaims(std::move(claims), provenance, generation); + ASSERT_TRUE(result.has_value()) << result.error(); +} + +ParserRouteRequest scalarRequest(std::string schema_digest = schemaDigest('a')) { + return { + .encoding = "protobuf", + .type_name = ".foxglove.PointCloud", + .schema_digest = std::move(schema_digest), + .route = ParserRoute::kScalar, + }; +} + +size_t traceCount(const ParserRouteResolution& result, ParserSelectionTraceKind kind) { + return static_cast( + std::count_if(result.trace.begin(), result.trace.end(), [&](const auto& entry) { return entry.kind == kind; })); +} + +TEST(ParserRouteResolver, ExactClaimsOutrankWildcardRegardlessOfTierAndPriority) { + ParserClaimCatalog catalog; + admit( + catalog, + { + claim("wildcard", "wild", "*", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, 1000, ParserClaimProvenance::kBundled), + }); + admit( + catalog, { + claim( + "exact", "exact", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, -1000, + ParserClaimProvenance::kFolderDrop), + }); + + ParserRouteResolver resolver; + const auto candidates = resolver.orderedCandidates(scalarRequest(), catalog, {}); + + ASSERT_TRUE(candidates.has_value()) << candidates.error(); + ASSERT_EQ(candidates->size(), 2U); + EXPECT_EQ((*candidates)[0].claim.provider_id, "exact"); + EXPECT_EQ((*candidates)[1].claim.provider_id, "wildcard"); +} + +TEST(ParserRouteResolver, ProvenanceBeatsPriorityAndPriorityBreaksTiesWithinTier) { + ParserClaimCatalog catalog; + admit( + catalog, {claim( + "folder-high", "c", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, 1000, + ParserClaimProvenance::kFolderDrop)}); + admit( + catalog, {claim( + "market-high", "b", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, 1000, + ParserClaimProvenance::kMarketplace)}); + admit( + catalog, {claim( + "bundled-low", "a", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, -1000, + ParserClaimProvenance::kBundled)}); + admit( + catalog, {claim( + "market-low", "d", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1, -5, + ParserClaimProvenance::kMarketplace)}); + + ParserRouteResolver resolver; + const auto candidates = resolver.orderedCandidates(scalarRequest(), catalog, {}); + + ASSERT_TRUE(candidates.has_value()) << candidates.error(); + ASSERT_EQ(candidates->size(), 4U); + EXPECT_EQ((*candidates)[0].claim.provider_id, "bundled-low"); + EXPECT_EQ((*candidates)[1].claim.provider_id, "market-high"); + EXPECT_EQ((*candidates)[2].claim.provider_id, "market-low"); + EXPECT_EQ((*candidates)[3].claim.provider_id, "folder-high"); +} + +TEST(ParserRouteResolver, StableIdentityBreaksEqualPolicyRanksLexicographically) { + ParserClaimCatalog catalog; + admit( + catalog, { + claim("provider-b", "claim-a"), + claim("provider-a", "claim-z"), + claim("provider-a", "claim-a"), + }); + + ParserRouteResolver resolver; + const auto candidates = resolver.orderedCandidates(scalarRequest(), catalog, {}); + + ASSERT_TRUE(candidates.has_value()) << candidates.error(); + ASSERT_EQ(candidates->size(), 3U); + EXPECT_EQ(candidates->at(0).claim.claim_id, "claim-a"); + EXPECT_EQ(candidates->at(1).claim.claim_id, "claim-z"); + EXPECT_EQ(candidates->at(2).claim.provider_id, "provider-b"); +} + +TEST(ParserRouteResolver, SchemaDigestAllowListsFilterCandidates) { + ParserClaimCatalog catalog; + auto supported = claim("supported", "supported"); + supported.schema_digests = {schemaDigest('a')}; + auto wrong = claim("wrong", "wrong"); + wrong.schema_digests = {schemaDigest('b')}; + admit(catalog, {supported, wrong}); + + ParserRouteResolver resolver; + const auto candidates = resolver.orderedCandidates(scalarRequest(), catalog, {}); + + ASSERT_TRUE(candidates.has_value()) << candidates.error(); + ASSERT_EQ(candidates->size(), 1U); + EXPECT_EQ(candidates->front().claim.provider_id, "supported"); +} + +TEST(ParserRouteResolver, DeclineAndErrorAdvanceAndRemainDistinctInTraceAndDiagnostics) { + std::vector diagnostics; + ParserRouteResolver resolver(recordInto(diagnostics)); + ParserClaimCatalog catalog; + admit( + catalog, { + claim("a-decline", "claim"), + claim("b-decline", "claim"), + claim("c-error", "claim"), + claim("d-accept", "claim"), + }); + std::vector probes; + + const auto result = resolver.resolve( + scalarRequest(), catalog, {}, + [](std::string_view) { return ParserProviderConfig{.json = R"({"enabled":true})", .digest = "config"}; }, + [&](const ParserProbeRequest& request) { + EXPECT_EQ(request.config_json, R"({"enabled":true})"); + probes.push_back(request.claim.provider_id); + if (request.claim.provider_id == "a-decline" || request.claim.provider_id == "b-decline") { + return probeDecision(ParserProbeOutcome::kDecline, "unsupported schema"); + } + if (request.claim.provider_id == "c-error") { + return probeDecision(ParserProbeOutcome::kError, "classification failed"); + } + return probeDecision(ParserProbeOutcome::kAccept, {}, std::make_shared(7)); + }); + + ASSERT_TRUE(result.has_value()) << result.error(); + EXPECT_EQ(result->status, ParserRouteResolutionStatus::kSelected); + ASSERT_TRUE(result->winner.has_value()); + EXPECT_EQ(result->winner->provider_id, "d-accept"); + EXPECT_EQ(probes, (std::vector{"a-decline", "b-decline", "c-error", "d-accept"})); + EXPECT_EQ(traceCount(*result, ParserSelectionTraceKind::kProbeDecline), 2U); + EXPECT_EQ(traceCount(*result, ParserSelectionTraceKind::kProbeError), 1U); + EXPECT_EQ(traceCount(*result, ParserSelectionTraceKind::kProbeAccept), 1U); + EXPECT_NE(result->retained_instance, nullptr); + + ASSERT_EQ(diagnostics.size(), 2U); + const auto decline_diagnostic = std::find_if(diagnostics.begin(), diagnostics.end(), [](const Diagnostic& entry) { + return entry.message.find("declined") != std::string::npos; + }); + const auto error_diagnostic = std::find_if(diagnostics.begin(), diagnostics.end(), [](const Diagnostic& entry) { + return entry.message.find("error") != std::string::npos; + }); + ASSERT_NE(decline_diagnostic, diagnostics.end()); + ASSERT_NE(error_diagnostic, diagnostics.end()); + EXPECT_EQ(decline_diagnostic->level, DiagnosticLevel::kInfo); + EXPECT_NE(decline_diagnostic->message.find("2 candidate"), std::string::npos); + EXPECT_EQ(error_diagnostic->level, DiagnosticLevel::kError); +} + +TEST(ParserRouteResolver, ScalarPinFailsClosedWithoutDarkeningUnpinnedObjectRoute) { + ParserClaimCatalog catalog; + admit( + catalog, + { + claim( + "pinned", "both", "foxglove.PointCloud", PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1), + claim( + "fallback", "both", "foxglove.PointCloud", + PJ_PARSER_ROUTE_FLAG_SCALAR_V1 | PJ_PARSER_ROUTE_FLAG_OBJECT_V1), + }); + ParserRouteResolver resolver; + const ParserRoutePins pins{.scalar_provider = "pinned", .object_provider = std::nullopt}; + std::vector probes; + auto probe = [&](const ParserProbeRequest& request) { + probes.push_back(request.claim.provider_id); + return probeDecision( + request.claim.provider_id == "pinned" ? ParserProbeOutcome::kDecline : ParserProbeOutcome::kAccept); + }; + + const auto scalar = resolver.resolve(scalarRequest(), catalog, pins, {}, probe); + ASSERT_TRUE(scalar.has_value()) << scalar.error(); + EXPECT_EQ(scalar->status, ParserRouteResolutionStatus::kPinnedProviderRejected); + EXPECT_FALSE(scalar->winner.has_value()); + EXPECT_EQ(probes, (std::vector{"pinned"})); + + auto object_request = scalarRequest(); + object_request.route = ParserRoute::kObject; + const auto object = resolver.resolve(object_request, catalog, pins, {}, probe); + ASSERT_TRUE(object.has_value()) << object.error(); + EXPECT_EQ(object->status, ParserRouteResolutionStatus::kSelected); + ASSERT_TRUE(object->winner.has_value()); + EXPECT_EQ(object->winner->provider_id, "fallback"); +} + +TEST(ParserRouteResolver, ObjectPinFailureDoesNotAffectScalarRoute) { + ParserClaimCatalog catalog; + admit(catalog, {claim("available", "scalar")}); + ParserRouteResolver resolver; + const ParserRoutePins pins{.scalar_provider = std::nullopt, .object_provider = "missing"}; + auto accept = [](const ParserProbeRequest&) { return probeDecision(ParserProbeOutcome::kAccept); }; + + auto object_request = scalarRequest(); + object_request.route = ParserRoute::kObject; + const auto object = resolver.resolve(object_request, catalog, pins, {}, accept); + ASSERT_TRUE(object.has_value()) << object.error(); + EXPECT_EQ(object->status, ParserRouteResolutionStatus::kPinnedProviderUnavailable); + + const auto scalar = resolver.resolve(scalarRequest(), catalog, pins, {}, accept); + ASSERT_TRUE(scalar.has_value()) << scalar.error(); + EXPECT_EQ(scalar->status, ParserRouteResolutionStatus::kSelected); + ASSERT_TRUE(scalar->winner.has_value()); + EXPECT_EQ(scalar->winner->provider_id, "available"); +} + +TEST(ParserRouteResolver, TieBreakDiagnosticIsEmittedOnlyOnce) { + std::vector diagnostics; + ParserRouteResolver resolver(recordInto(diagnostics)); + ParserClaimCatalog catalog; + admit(catalog, {claim("b-provider", "claim"), claim("a-provider", "claim")}); + size_t calls = 0; + auto accept = [&](const ParserProbeRequest&) { + ++calls; + return probeDecision(ParserProbeOutcome::kAccept); + }; + + const auto first = resolver.resolve(scalarRequest(), catalog, {}, {}, accept); + const auto second = resolver.resolve(scalarRequest(), catalog, {}, {}, accept); + + ASSERT_TRUE(first.has_value()) << first.error(); + ASSERT_TRUE(second.has_value()) << second.error(); + ASSERT_TRUE(first->winner.has_value()); + EXPECT_EQ(first->winner->provider_id, "a-provider"); + EXPECT_EQ(traceCount(*first, ParserSelectionTraceKind::kAmbiguityTieBreak), 1U); + EXPECT_EQ(traceCount(*second, ParserSelectionTraceKind::kAmbiguityTieBreak), 0U); + EXPECT_EQ(traceCount(*second, ParserSelectionTraceKind::kCacheHit), 1U); + EXPECT_EQ(calls, 1U); + ASSERT_EQ(diagnostics.size(), 1U); + EXPECT_NE(diagnostics.front().message.find("ambiguous"), std::string::npos); +} + +TEST(ParserRouteResolver, ProbeCacheKeysAndExplicitInvalidationsAreObserved) { + ParserClaimCatalog catalog; + admit(catalog, {claim("provider", "claim")}, 10); + ParserRouteResolver resolver; + std::map configs{{"provider", "config-a"}}; + size_t calls = 0; + auto lookup = [&](std::string_view provider) { + const auto digest = configs.at(std::string(provider)); + return ParserProviderConfig{.json = "{}", .digest = digest}; + }; + auto accept = [&](const ParserProbeRequest&) { + ++calls; + return probeDecision(ParserProbeOutcome::kAccept, {}, std::make_shared(calls)); + }; + + ASSERT_TRUE(resolver.resolve(scalarRequest(), catalog, {}, lookup, accept).has_value()); + auto cached = resolver.resolve(scalarRequest(), catalog, {}, lookup, accept); + ASSERT_TRUE(cached.has_value()) << cached.error(); + EXPECT_EQ(calls, 1U); + EXPECT_EQ(traceCount(*cached, ParserSelectionTraceKind::kCacheHit), 1U); + + ASSERT_TRUE(resolver.resolve(scalarRequest(schemaDigest('b')), catalog, {}, lookup, accept).has_value()); + EXPECT_EQ(calls, 2U); + + configs["provider"] = "config-b"; + ASSERT_TRUE(resolver.resolve(scalarRequest(), catalog, {}, lookup, accept).has_value()); + EXPECT_EQ(calls, 3U); + resolver.invalidateProviderConfig("provider"); + ASSERT_TRUE(resolver.resolve(scalarRequest(), catalog, {}, lookup, accept).has_value()); + EXPECT_EQ(calls, 4U); + + resolver.invalidatePins(); + ASSERT_TRUE(resolver.resolve(scalarRequest(), catalog, {}, lookup, accept).has_value()); + EXPECT_EQ(calls, 5U); + resolver.invalidateCatalog(); + ASSERT_TRUE(resolver.resolve(scalarRequest(), catalog, {}, lookup, accept).has_value()); + EXPECT_EQ(calls, 6U); +} + +TEST(ParserRouteResolver, ProviderGenerationParticipatesInProbeCacheKey) { + ParserClaimCatalog first_catalog; + ParserClaimCatalog second_catalog; + admit(first_catalog, {claim("provider", "claim")}, 1); + admit(second_catalog, {claim("provider", "claim")}, 2); + ParserRouteResolver resolver; + size_t calls = 0; + auto accept = [&](const ParserProbeRequest&) { + ++calls; + return probeDecision(ParserProbeOutcome::kAccept); + }; + + ASSERT_TRUE(resolver.resolve(scalarRequest(), first_catalog, {}, {}, accept).has_value()); + ASSERT_TRUE(resolver.resolve(scalarRequest(), second_catalog, {}, {}, accept).has_value()); + EXPECT_EQ(calls, 2U); + EXPECT_EQ(resolver.probeCacheSize(), 2U); +} + +TEST(ParserRouteResolver, SameProviderExactDeclineDoesNotPoisonWildcardProbeCache) { + ParserClaimCatalog catalog; + admit( + catalog, { + claim("provider", "exact", "foxglove.PointCloud"), + claim("provider", "wildcard", "*"), + }); + ParserRouteResolver resolver; + std::vector probed; + const auto result = resolver.resolve(scalarRequest(), catalog, {}, {}, [&](const ParserProbeRequest& request) { + probed.push_back(request.claim.claim_id); + return probeDecision( + request.claim.claim_id == "exact" ? ParserProbeOutcome::kDecline : ParserProbeOutcome::kAccept); + }); + + ASSERT_TRUE(result.has_value()) << result.error(); + EXPECT_EQ(result->status, ParserRouteResolutionStatus::kSelected); + ASSERT_TRUE(result->winner.has_value()); + EXPECT_EQ(result->winner->claim_id, "wildcard"); + EXPECT_EQ(probed, (std::vector{"exact", "wildcard"})); + EXPECT_EQ(traceCount(*result, ParserSelectionTraceKind::kCacheHit), 0U); +} + +} // namespace +} // namespace PJ