SQL: struct support#586
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
| :learning-objective-2: Query nested fields using ROW field-access syntax | ||
| :learning-objective-3: Recognize and resolve cyclic-reference errors | ||
|
|
||
| When a glossterm:topic[]'s schema includes nested Protobuf or Avro message types, you can map those nested structures as SQL `ROW` columns instead of opaque JSON. This makes nested fields queryable by name, includable in projections, and usable in `WHERE`, `GROUP BY`, and `ORDER BY` clauses, without parsing JSON at query time. |
There was a problem hiding this comment.
That may be a nitpick, but stating that we are mapping as SQL ROW columns is not entirely true.
In PostgreSQL, a ROW is an anonymous record, in which you cannot explicitly set the sub-field names (they contain some generic f1, f2, f<n>... names that you cannot change).
What we do in the COMPOUND mapping is we actually create a User-Defined type, and set the names of the fields according to the schema.
Not sure if that's something that we want to explicitly state here, or maybe the ROW meaning here is something other than PostgreSQL ROW.
| } | ||
| ---- | ||
|
|
||
| Redpanda SQL maps the table with three columns: `order_id` (text), `customer` (a `ROW` with fields `customer_id`, `name`, and `region`), and `amount` (double precision). |
There was a problem hiding this comment.
ditto here:
a
ROWwith fields
We may also say something along the lines of:
a structure/UDT with fields
|
|
||
| Redpanda SQL maps the table with three columns: `order_id` (text), `customer` (a `ROW` with fields `customer_id`, `name`, and `region`), and `amount` (double precision). | ||
|
|
||
| TIP: To map nested structures as JSON instead, use `struct_mapping_policy = 'JSON'`. `JSON` is the default, and it is the only option that supports recursive (cyclic) types. See <<handle-recursive-cyclic-schemas, Handle recursive (cyclic) schemas>>. |
There was a problem hiding this comment.
I believe JSON will not be the default. That's something that is being changed right now, and we want COMPUND to be the default.
| (1 row) | ||
| ---- | ||
|
|
||
| === Use implicit tuple syntax |
There was a problem hiding this comment.
I'm late to the party, as this was not modified in this PR :D I believe it's worth noting that the implicit tuple syntax works only when there are two or more expressions
|
|
||
| == Check for NULL | ||
|
|
||
| ROW values support `IS NULL` and `IS NOT NULL`. |
There was a problem hiding this comment.
The behavior of IS NULL/IS NOT NULL is a bit different for structs than it is for "scalar" columns, there is a short description here https://redpandadata.atlassian.net/wiki/spaces/OXLA/pages/1481413789/Structures+-+implementation+overview#2.5.-IS-NULL-%2F-IS-NOT-NULL
I think it's worth noting it.
|
|
||
| * Enable Redpanda SQL on your Redpanda Bring Your Own Cloud (BYOC) cluster. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL]. | ||
| * Connect to Redpanda SQL with `psql` or another PostgreSQL client. See xref:sql:connect-to-sql/index.adoc[Connect to Redpanda SQL]. | ||
| * The topic has a schema (Protobuf or Avro) registered in glossterm:schema-registry[Schema Registry]. The schema includes one or more nested message types. |
There was a problem hiding this comment.
Here, it is mentioned that Protobuf or Avro schema must be used, but it should work for JSON schemas as well.
2 participants
Description
This pull request introduces comprehensive documentation improvements for working with nested fields in Redpanda SQL, focusing on mapping nested Protobuf or Avro structures as SQL ROW columns and querying them directly. It clarifies the use of the
struct_mapping_policyoption, expands the reference for theROWdata type, and adds a dedicated how-to guide for querying nested fields.New documentation and feature explanations:
query-nested-fields.adoc, detailing how to map topics with nested schemas as SQL tables usingstruct_mapping_policy = 'COMPOUND', how to query nested fields with ROW syntax, and how to handle recursive (cyclic) schemas.nav.adoc) to include the new "Query Topics with Nested Fields" guide.Improvements to ROW data type documentation:
ROWdata type reference to document field access (by position and name), wildcard projection, lexicographic comparison, NULL checks, text conversion, and usage inGROUP BY,ORDER BY, andJOINclauses.ROWtype summary to mention its support for field access, comparisons, and use in query clauses.Clarifications to CREATE TABLE options:
struct_mapping_policyoption in theCREATE TABLEdocumentation, emphasizing thatCOMPOUNDmaps nested structures to SQL ROW columns and noting that cyclic types are only supported inJSONmode.Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline:
Page previews
Reference > Redpanda SQL Reference > Data Types > Row
Redpanda SQL > Query Data > Query Topics with Nested Fields
Checks