Skip to content

feat: support filtering on table calculations#24

Merged
jpetey75 merged 3 commits into
mainfrom
feat/table-calculation-filters
Jun 25, 2026
Merged

feat: support filtering on table calculations#24
jpetey75 merged 3 commits into
mainfrom
feat/table-calculation-filters

Conversation

@jpetey75

@jpetey75 jpetey75 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Closes #21

Problem

The SDK's filter system was dimension-only — there was no way to filter on table calculations, even though the API supports it via filters.tableCalculations.

Change

  • TableCalculation(name, sql, display_name=None, type="number") — a SQL table calculation, with the same comparison operators (==, >, between, etc.) and helpers as Dimension.
  • TableCalculationFilter — accepts a TableCalculation or a calc-name string; serializes as {target: {fieldId}, operator, values}.
  • CompositeFilter.to_dict() now routes dimension vs table-calc rules into filters.dimensions / filters.tableCalculations respectively. The tableCalculations key is omitted when unused, so existing dimension-only payloads are unchanged.
  • Query.table_calculations(...) builder method + model.query(table_calculations=...) kwarg.
  • Extracted a shared _FieldFilterMixin for the &/| operators rather than duplicating them across filter classes.
from lightdash import TableCalculation

profit_ratio = TableCalculation(name="profit_ratio", sql="${orders.profit} / ${orders.revenue}")

query = (
    model.query()
    .metrics(model.metrics.revenue, model.metrics.profit)
    .table_calculations(profit_ratio)
    .filter(profit_ratio > 0.2)
)

Verification

Shapes were checked against the authoritative lightdash/common source types rather than guessed:

  • Filters = { dimensions?, metrics?, tableCalculations? }, each a FilterGroup → confirms the filters.tableCalculations routing.
  • FilterRule = { target: {fieldId}, operator, values } (server injects id) → confirms the rule shape.
  • SqlTableCalculation = { name, displayName, sql, type } → confirms TableCalculation.to_dict().

End-to-end verified against a live instance (see comment): defined a calc, filtered calc > threshold, and confirmed the server returned exactly the rows a client-side filter would. This surfaced and fixed a real bug — an untyped calc is treated as string by the API and rejects numeric operators, hence type now defaults to "number".

Known limitation — OR across field types

Lightdash's Filters shape has separate groups (dimensions / metrics / tableCalculations) that are AND-ed together at the top level. So an OR that mixes a dimension and a table-calc rule — e.g. (dim == "USA") | (calc > 0.2) — cannot be expressed faithfully; it serializes as two independent groups that the server ANDs. OR within a single field type works as expected. This is an API constraint, now documented in the SDK guide.

Tests

27 new tests in tests/test_table_calculations.py (calc serialization, operators, filters.tableCalculations routing, mixed dimension+calc filters, builder integration) plus an env-gated E2E acceptance test. Full unit suite green.

🤖 Generated with Claude Code

@jpetey75

Copy link
Copy Markdown
Contributor Author

End-to-end verification (live instance) — found & fixed a real bug

Ran the SDK against a live Lightdash instance with a real token. The first attempt failed with a 400, which source-reading alone hadn't caught:

CompileError: No function has been implemented to render SQL for the filter
operator "greaterThan" on "pv_copy" field of type "string"

Root cause: a table calculation with no declared type is treated as a string by the API, so numeric operators (>, between, …) won't compile. The filters.tableCalculations routing and rule shape were correct — the calc definition was just missing its type.

Fix (latest commit): TableCalculation now has a type field defaulting to "number" (emitted in to_dict(), matching TableCalculationBase). Override for non-numeric calcs (type="string" etc.).

After the fix — verified end-to-end: defined a calc, filtered calc > threshold, and confirmed the server returned exactly the rows a client-side filter would (every returned row satisfies the predicate; count matches). Added as a permanent env-gated acceptance test (test_query_table_calculation_filter), which passes live.

So this PR is now confirmed working end-to-end, not just shape-verified — and the earlier caveat about not being able to run E2E is resolved.

Base automatically changed from feat/multiple-filters-same-field to main June 24, 2026 20:02
jpetey75 and others added 2 commits June 24, 2026 20:24
Add a TableCalculation type and TableCalculationFilter so queries can filter
on calculated values. Table-calc filters serialize under
`filters.tableCalculations` (a sibling of `filters.dimensions`), matching the
Lightdash `Filters` type; the calculation itself serializes as a
`SqlTableCalculation` (`{name, displayName, sql}`).

- `TableCalculation(name, sql, display_name=None)` with the same comparison
  operators / helper methods as `Dimension`.
- `TableCalculationFilter` accepts a `TableCalculation` or a calc name string.
- `CompositeFilter.to_dict()` routes dimension vs table-calc rules into their
  respective keys; the `tableCalculations` key is omitted when unused so
  existing dimension-only payloads are byte-for-byte unchanged.
- `Query.table_calculations()` builder method + `model.query(table_calculations=...)`.
- Extracted a shared `_FieldFilterMixin` for the `&`/`|` operators instead of
  duplicating them across filter classes.

Shapes verified against lightdash/common source types and the live API's
filter-rule parser.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
End-to-end testing against a live instance surfaced a 400 CompileError:
filtering a table calculation with a numeric operator failed because an
untyped calc is treated as a "string" field by the API
("No function ... for the filter operator 'greaterThan' on ... field of type
'string'").

- `TableCalculation` gains a `type` field (defaults to "number"); `to_dict()`
  now emits it as part of the SqlTableCalculation, matching TableCalculationBase.
- Added an env-gated acceptance test (`test_query_table_calculation_filter`)
  that defines a calc, filters on it, and asserts the server applied the filter
  (every returned row satisfies the predicate and the count matches a
  client-side computation). Verified live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jpetey75 jpetey75 force-pushed the feat/table-calculation-filters branch from 08b5a20 to 4973701 Compare June 25, 2026 01:37
Lightdash AND-combines the dimensions/metrics/tableCalculations filter
groups at the top level, so an OR mixing a dimension and a table-calc
rule cannot be expressed faithfully. Document this and soften the
"combined freely" wording to AND-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFPouw17Yc7W8kQ4tLZSrp
@jpetey75 jpetey75 merged commit 9e35685 into main Jun 25, 2026
2 checks passed
@jpetey75 jpetey75 deleted the feat/table-calculation-filters branch June 25, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support filtering on table calculations

2 participants