Document Tiered Storage v2 (26.2)

docs-data/property-overrides.json

@@ -779,8 +779,21 @@
       "config_scope": "cluster",
       "version": "v26.1.1"
     },
+    "default_redpanda_storage_mode_tiered_impl": {
+      "description": "Default implementation of the `tiered` storage mode for newly created topics. When the `redpanda.storage.mode` topic property is set to `tiered`, this property determines whether the topic uses Tiered Storage v1 (`tiered_v1`) or Tiered Storage v2 (`tiered_v2`). The implementation of each topic is reported by the read-only `redpanda.storage.mode.impl` topic property.\n\nChanging this property affects only topics created after the change. Existing topics keep the Tiered Storage version they were created with.",
+      "accepted_values": [
+        "tiered_v1",
+        "tiered_v2"
+      ],
+      "example": "`tiered_v2`",
+      "related_topics": [
+        "xref:manage:tiered-storage.adoc#tiered-storage-versions[]"
+      ],
+      "version": "v26.2.1",
+      "config_scope": "cluster"
+    },
     "delete.retention.ms": {
-      "description": "The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.\n\nIf you have enabled Tiered Storage and set <<redpandaremoteread,`redpanda.remote.read`>> or <<redpandaremotewrite,`redpanda.remote.write`>> for the topic, you cannot enable tombstone removal.\n\nIf both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.\n\nThis property supports three states:\n\n* Positive value: Sets the milliseconds to retain tombstone records before removal.\n* 0: Tombstone records are immediately eligible for removal.\n* Negative value: Disables tombstone removal entirely for this topic.",
+      "description": "The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.\n\nFor topics that use Tiered Storage v1, if you have enabled Tiered Storage and set <<redpandaremoteread,`redpanda.remote.read`>> or <<redpandaremotewrite,`redpanda.remote.write`>> for the topic, you cannot enable tombstone removal.\nifndef::env-cloud[]\nThis restriction does not apply to topics that use xref:manage:tiered-storage.adoc#tiered-storage-versions[Tiered Storage v2], available starting in Redpanda v26.2.\nendif::[]\nifdef::env-cloud[]\nThis restriction does not apply to topics that use Tiered Storage v2, available starting in Redpanda v26.2.\nendif::[]\n\nIf both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.\n\nThis property supports three states:\n\n* Positive value: Sets the milliseconds to retain tombstone records before removal.\n* 0: Tombstone records are immediately eligible for removal.\n* Negative value: Disables tombstone removal entirely for this topic.",
       "related_topics": [
         "xref:reference:properties/cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`]",
         "xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal]"
@@ -1831,7 +1844,7 @@
       "config_scope": "topic"
     },
     "redpanda.storage.mode": {
-      "description": "The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage.\n\nAccepted values:\n\n* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings.\nifndef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic.\n* `cloud`: Topic data is stored in object storage using the xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] architecture. Local storage is used only as a write buffer.\nendif::[]\nifdef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables Tiered Storage for the topic.\n* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer.\nendif::[]\n* `unset`: Specifies that the topic's storage mode is unset, regardless of the cluster default. The topic may still have Tiered Storage enabled through the legacy properties `redpanda.remote.read` and `redpanda.remote.write`.\n\nThis property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics.",
+      "description": "The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage.\n\nAccepted values:\n\n* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings.\nifndef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic.\n* `cloud`: Topic data is stored in object storage using the xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] architecture. Local storage is used only as a write buffer.\nendif::[]\nifdef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables Tiered Storage for the topic.\n* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer.\nendif::[]\n* `unset`: Specifies that the topic's storage mode is unset, regardless of the cluster default. The topic may still have Tiered Storage enabled through the legacy properties `redpanda.remote.read` and `redpanda.remote.write`.\n\nStarting in Redpanda v26.2, the `tiered` value is an alias that resolves to Tiered Storage v1 or v2 when the topic is created, according to the config_ref:default_redpanda_storage_mode_tiered_impl,true,properties/cluster-properties[] cluster property. The resolved version is reported by the read-only <<redpandastoragemodeimpl,`redpanda.storage.mode.impl`>> property, and the storage mode of a topic displays as `tiered` for both versions.\n\nThis property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics.",
       "related_topics": [
         "self-managed-only: xref:manage:tiered-storage.adoc[Tiered Storage]",
         "self-managed-only: xref:develop:manage-topics/cloud-topics.adoc[Manage Cloud Topics]"
@@ -1840,6 +1853,21 @@
       "category": "tiered-storage",
       "version": "v26.1.1"
     },
+    "redpanda.storage.mode.impl": {
+      "description": "The exact implementation of the topic's storage mode. Topics with the `tiered` storage mode report `tiered_v1` (Tiered Storage v1) or `tiered_v2` (Tiered Storage v2); topics with other storage modes mirror the value of <<redpandastoragemode,`redpanda.storage.mode`>>.\n\nThis property is read-only after topic creation. To select the Tiered Storage version for a topic explicitly, set this property together with a matching `redpanda.storage.mode` value when you create the topic (both `tiered_v1` and `tiered_v2` correspond to the `tiered` mode). You can omit `redpanda.storage.mode` only if this property matches the version selected by the cluster default. To change a topic's storage mode after creation, alter `redpanda.storage.mode` instead.\n\nIf this property is not set at topic creation, the `tiered` storage mode resolves to the version selected by the config_ref:default_redpanda_storage_mode_tiered_impl,true,properties/cluster-properties[] cluster property.",
+      "accepted_values": [
+        "local",
+        "tiered_v1",
+        "tiered_v2",
+        "cloud",
+        "unset"
+      ],
+      "related_topics": [
+        "xref:manage:tiered-storage.adoc#tiered-storage-versions[]"
+      ],
+      "version": "v26.2.1",
+      "config_scope": "topic"
+    },
     "redpanda.value.schema.id.validation": {
       "description": "Enable validation of the schema ID for values on a record. When enabled, Redpanda validates that the schema ID encoded in the record's value is registered in the Schema Registry according to the configured subject name strategy.",
       "related_topics": [
@@ -2199,7 +2227,7 @@
       "config_scope": "cluster"
     },
     "tombstone_retention_ms": {
-      "description": "The retention time for tombstone records in a compacted topic. Cannot be enabled at the same time as any of `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, or `cloud_storage_enable_remote_write`. A typical default setting is `86400000`, or 24 hours.",
+      "description": "The retention time for tombstone records in a compacted topic. For Tiered Storage v1, cannot be enabled at the same time as any of `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, or `cloud_storage_enable_remote_write`. This restriction does not apply to topics that use xref:manage:tiered-storage.adoc#tiered-storage-versions[Tiered Storage v2], available starting in Redpanda v26.2. A typical default setting is `86400000`, or 24 hours.",
       "related_topics": [
         "xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal]"
       ],

modules/develop/pages/manage-topics/config-topics.adoc

@@ -76,7 +76,7 @@ Starting in Redpanda v26.1, you can set the `redpanda.storage.mode` topic proper
 | Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of `redpanda.remote.read` and `redpanda.remote.write` values.
 
 | `tiered`
-| Data is stored on local disk and uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic regardless of `redpanda.remote.read` and `redpanda.remote.write` values.
+| Data is stored on local disk and uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic regardless of `redpanda.remote.read` and `redpanda.remote.write` values. Starting in Redpanda v26.2, Tiered Storage is available in two versions (`tiered_v1` and `tiered_v2`). See xref:manage:tiered-storage.adoc#tiered-storage-versions[Choose a Tiered Storage version].
 
 | `cloud`
 | Data is stored durably in object storage using the glossterm:Cloud Topic[,Cloud Topics] architecture. Local storage is used only as a write buffer. See xref:develop:manage-topics/cloud-topics.adoc[].
@@ -91,6 +91,15 @@ rpk topic create <topic-name> -c redpanda.storage.mode=tiered
 
 When `redpanda.storage.mode` is set to `local`, `tiered`, or `cloud`, the `redpanda.remote.read` and `redpanda.remote.write` topic properties have no effect on the topic.
 
+Starting in Redpanda v26.2, the `tiered` value is an alias that resolves to one of two Tiered Storage versions when the topic is created. The xref:reference:cluster-properties.adoc#default_redpanda_storage_mode_tiered_impl[`default_redpanda_storage_mode_tiered_impl`] cluster property selects the version (`tiered_v1` by default). To select the version explicitly for a single topic, set both `redpanda.storage.mode` and `redpanda.storage.mode.impl` when you create the topic:
+
+[,bash]
+----
+rpk topic create <topic-name> -c redpanda.storage.mode=tiered -c redpanda.storage.mode.impl=tiered_v2
+----
+
+The `redpanda.storage.mode.impl` property reports the exact implementation of every topic's storage mode and is read-only after topic creation. For details, see xref:manage:tiered-storage.adoc#tiered-storage-versions[Choose a Tiered Storage version].
+
 To apply a default storage mode to all new topics in a cluster, set the `default_redpanda_storage_mode` cluster property:
 
 [,bash]
@@ -151,33 +160,57 @@ You can change a topic's `redpanda.storage.mode` after creation, with the follow
 | From | To | Permitted | Notes
 
 | `local`
-| `tiered`
+| `tiered` (v1)
 | Yes
 | Enables Tiered Storage for the topic. Object storage must be configured.
 
-| `tiered`
+| `tiered` (v1)
 | `local`
 | With caution
 | Disables object storage uploads. Redpanda strongly recommends against repeatedly toggling this setting, as it can result in data gaps in Tiered Storage.
 
-| Any
 | `cloud`
+| `tiered` (v2)
+| Yes
+| Starting in Redpanda v26.2. All brokers must run v26.2 or later.
+
+| `tiered` (v2)
+| `cloud`
+| Yes
+| Starting in Redpanda v26.2.
+
+| `tiered` (v1)
+| `tiered` (v2)
+| No
+| Topics cannot be converted between Tiered Storage versions, in either direction.
+
+| `local`
+| `tiered` (v2)
 | No
-| Cloud Topics can only be set at topic creation time.
+|
 
+| Any other mode
 | `cloud`
-| Any
 | No
-| A Cloud Topic cannot be converted to a local or Tiered Storage topic.
+| Cloud Topics can otherwise only be set at topic creation time.
+
+| `cloud`
+| `local` or `tiered` (v1)
+| No
+| A Cloud Topic cannot be converted to a local or Tiered Storage v1 topic.
 |===
 
+When you change the storage mode, you always set the `redpanda.storage.mode` property. The `tiered` value resolves to the version selected by the xref:reference:cluster-properties.adoc#default_redpanda_storage_mode_tiered_impl[`default_redpanda_storage_mode_tiered_impl`] cluster property, so, for example, converting a `cloud` topic to Tiered Storage v2 requires `default_redpanda_storage_mode_tiered_impl=tiered_v2`. The `redpanda.storage.mode.impl` property cannot be changed after topic creation. See xref:manage:tiered-storage.adoc#tiered-storage-versions[Choose a Tiered Storage version].
+
 For example, to transition an existing local topic to Tiered Storage:
 
 [,bash]
 ----
 rpk topic alter-config <topic-name> --set redpanda.storage.mode=tiered
 ----
 
+NOTE: This example requires xref:reference:cluster-properties.adoc#default_redpanda_storage_mode_tiered_impl[`default_redpanda_storage_mode_tiered_impl`] to be `tiered_v1` (the default). If the cluster default is `tiered_v2`, the request is rejected, because a `local` topic cannot be converted to Tiered Storage v2.
+
 endif::[]
 
 
@@ -283,6 +316,7 @@ redpanda.remote.delete        true                           DEFAULT_CONFIG
 redpanda.remote.read          false                          DEFAULT_CONFIG
 redpanda.remote.write         false                          DEFAULT_CONFIG
 redpanda.storage.mode         unset                          DEFAULT_CONFIG
+redpanda.storage.mode.impl    unset                          DEFAULT_CONFIG
 retention.bytes               -1                             DEFAULT_CONFIG
 retention.local.target.bytes  -1                             DEFAULT_CONFIG
 retention.local.target.ms     86400000                       DEFAULT_CONFIG

modules/develop/partials/cloud-topics-limitations.adoc

@@ -1,6 +1,6 @@
 // tag::limitations[]
 - Shadow links do not currently support Cloud Topics.
-- Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local or Tiered Storage. Conversely, existing topics created as local or Tiered Storage topics cannot be converted to Cloud Topics.
+- Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local storage or Tiered Storage v1. Conversely, existing topics created as local or Tiered Storage v1 topics cannot be converted to Cloud Topics. Starting in Redpanda v26.2, Cloud Topics can be converted to and from Tiered Storage v2 topics.
 // end::limitations[]
 
 // tag::latency-limitation[]

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

@@ -110,11 +110,13 @@ You can specify how long Redpanda keeps these tombstones for compacted topics us
 
 [NOTE]
 ====
-Redpanda does not currently remove tombstone records for compacted topics with Tiered Storage enabled. 
+Redpanda does not remove tombstone records for compacted topics that use Tiered Storage v1. Compacted topics that use xref:manage:tiered-storage.adoc#tiered-storage-versions[Tiered Storage v2], available starting in Redpanda v26.2, support tombstone removal, configured with `delete.retention.ms`.
 
-You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`.
+The following restrictions apply to Tiered Storage v1. They do not apply to topics that use Tiered Storage v2:
 
-On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`.
+* You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`.
+
+* On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`.
 ====
 
 To set the cluster-level tombstone retention policy, run the command: