diff --git a/modules/develop/pages/connect/connect-quickstart.adoc b/modules/develop/pages/connect/connect-quickstart.adoc index c628df50..4d5fc539 100644 --- a/modules/develop/pages/connect/connect-quickstart.adoc +++ b/modules/develop/pages/connect/connect-quickstart.adoc @@ -1,9 +1,22 @@ = Redpanda Connect Quickstart :description: Learn how to quickly start building data pipelines with Redpanda Connect. -The *Connect* page provides a wizard to create pipelines for streaming data into and out of Redpanda. The wizard populates the YAML configuration automatically, so you can get started quickly. +The *Connect* page lets you build data pipelines using a YAML editor. Add connectors and processors from the left pane and configure them in the editor. -TIP: Advanced users can skip directly to the *Edit pipeline* step in the wizard to configure the YAML file themselves. +[TIP] +==== +* Click any connector to learn more about it in the documentation. +* Use slash commands (`/`) in the YAML editor to insert existing variables, secrets, topics, and users, or to create new ones. + +==== + +== Prerequisites + +You must have a Redpanda Cloud account with a Serverless, Dedicated, or standard BYOC cluster. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^]. + +NOTE: Serverless clusters support up to 10 pipelines. + +== Quickstart pipelines This quickstart shows how Redpanda Connect can generate, transform, and handle streaming data end-to-end. It creates the following pipelines: @@ -12,7 +25,7 @@ This quickstart shows how Redpanda Connect can generate, transform, and handle s The *producer pipeline* uses the following Redpanda Connect components: -[cols="1,2,3"] +[cols="1,2,3"]a |=== |Component type |Component |Purpose @@ -61,29 +74,17 @@ The *consumer pipeline* uses the following Redpanda Connect components: |Catches errors |=== -== Prerequisites - -You must have a Redpanda Cloud account with a Serverless, Dedicated, or standard BYOC cluster. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^]. - -NOTE: Serverless clusters support up to 10 pipelines. - == Build a producer pipeline -Follow these steps to create the producer pipeline: - -. Go to the **Connect** page for your cluster to create a pipeline. +To create the producer pipeline: -. **Add an input**: Search for and select `generate` from the list of connectors. Click **Next**. +. Go to the *Connect* page for your cluster and click *Create a pipeline*. -. **Add an output**: Search for and select `redpanda` from the list of connectors. Click **Next**. +. Enter this name for the pipeline: `joke-generator-producer`. -. **Add a topic**: Create a new topic called `dad-jokes`. This is where Redpanda will store the generated messages. Click **Next**. +. In the left pane, click to add the `generate` input connector and the `redpanda` output connector. The YAML for these connectors appears in the editor. -. **Add permissions**: Create a new user called `connect`. Leave the rest of the default settings. The user will be created with the necessary permissions. Click **Next**. - -. **Edit pipeline**: -.. Enter this name for the pipeline: `joke-generator-producer`. -.. The **Configuration** section automatically populates the YAML with your selected components. Under *Connectors*, you can add the processors `log` and `catch` to log generated jokes and monitor for errors. However, for simplicity in this quickstart, replace the entire configuration with the following YAML, which includes the processors and the `mapping` for joke generation: +. Replace the generated YAML with the following configuration, which includes the `log` and `catch` processors and the `mapping` for joke generation: + [source,yaml] ---- @@ -151,17 +152,20 @@ output: ---- -.. Click **Create**. +. Click *Create*. + [NOTE] ==== -* Notice the `$\{REDPANDA_BROKERS}` xref:develop:connect/configuration/contextual-variables.adoc[contextual variable] in the configuration. This automatically references your cluster's bootstrap server address, so you can use it in any pipeline without hardcoding connection details. -* Notice `${secrets.KAFKA_USER_CONNECT}` and `${secrets.KAFKA_PASSWORD_CONNECT}`. These were generated when you created the `connect` user. +* Notice the `$\{REDPANDA_BROKERS}` xref:develop:connect/configuration/contextual-variables.adoc[contextual variable] in the configuration. This automatically references your cluster's bootstrap server address, so you can use it in any pipeline without hardcoding connection details. +* Notice `${secrets.KAFKA_USER_CONNECT}` and `${secrets.KAFKA_PASSWORD_CONNECT}`. These reference secrets that you can create using the slash command menu in the YAML editor or the *Security* page. * The Brave browser does not fully support code snippets. ==== ++ +TIP: In the YAML editor, you can type `/` to open the slash command menu. Use it to insert variables, secrets, topics, and users. You can select existing items or create new ones. + . Your pipeline details display, and after a few seconds, the status changes from **Starting** to **Running**. If you don't see this change, refresh the page. Once running, your pipeline generates a new joke every five seconds and writes the joke to your topic. + @@ -169,7 +173,7 @@ After a minute, select the pipeline and click **Stop** so you can examine the re === Review the pipeline logs -. Click the **Logs** tab to see the pipeline's activity log. +. On your pipeline page, switch to the **Live** mode to see the pipeline's activity log. The live mode loads new log messages as they come in. When Live mode is disabled, the log shows activity for from the past five hours. . Click through the log messages to see the startup sequence. For example, you'll see when the output becomes active: + @@ -206,24 +210,15 @@ After a minute, select the pipeline and click **Stop** so you can examine the re == Build a consumer pipeline -This pipeline rates the jokes that you generated in the first pipeline. Follow these steps to create the consumer pipeline: - -. On the **Connect** page for your cluster, click **Create pipeline**. - -. **Add an input**: Search for and select `redpanda` from the list of connectors. Click **Next**. +This pipeline rates the jokes that you generated in the first pipeline. To create the consumer pipeline: -. **Add an output**: Search for and select `drop` from the list of connectors. (Because this quickstart is just for testing, this output drops the message instead of sending it anywhere else. In a real scenario you'd replace the `drop` connector with your real destination.) Click **Next**. +. On the *Connect* page for your cluster, click *Create a pipeline*. -. **Add a topic**: Select the existing topic called `dad-jokes`. Click **Next**. +. Enter this name for the pipeline: `joke-generator-consumer`. -. **Add permissions**: -.. Select the existing user called `connect`. -.. Add a consumer group: Enter `dad-joke-raters` as the name for the consumer group. -.. Click **Next**. +. In the left pane, click to add the `redpanda` input connector and the `drop` output connector. (Because this quickstart is just for testing, this output drops the message instead of sending it anywhere else. In a real scenario you'd replace the `drop` connector with your real destination.) Their YAML appears in the editor. -. **Edit pipeline**: -.. Enter this name for the pipeline: `joke-generator-consumer`. -.. The **Configuration** section automatically populates the YAML with your selected components. To add the `bloblang`, `log`, and `catch` processors, replace the entire configuration with the following YAML. Bloblang is Redpanda Connect's scripting language used to add logic (for example, random ratings). +. Replace the generated YAML with the following configuration, which includes the `bloblang`, `log`, and `catch` processors. Bloblang is Redpanda Connect's scripting language used to add logic (for example, random ratings). + [source,yaml] @@ -317,7 +312,7 @@ output: ---- -.. Click **Create** to start your pipeline. +. Click *Create* to start your pipeline. + NOTE: This example explicitly includes several optional configuration fields for the `redpanda` input. They're shown here for demonstration purposes, so you can see a range of available settings. @@ -352,7 +347,7 @@ When you've finished experimenting with your data pipeline, you can delete the p == Next steps -* Try one of our xref:cookbooks:index.adoc[Redpanda Connect cookbooks]. +* Try one of the xref:cookbooks:index.adoc[Redpanda Connect cookbooks]. * Choose xref:develop:connect/components/about.adoc[connectors for your use case]. * Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline]. * Learn how to xref:develop:connect/configuration/monitor-connect.adoc[monitor a data pipeline on a BYOC or Dedicated cluster]. diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index 69c09576..566dc4cf 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -110,11 +110,11 @@ The message size setting on existing topics is not changed, but the message size === Redpanda Connect updates -Redpanda Connect provides a simplified xref:develop:connect/connect-quickstart.adoc[quickstart] experience in the UI that helps you to start building data pipelines. The quickstart wizard creates pipelines to stream data into and out of Redpanda, populating the required YAML configuration automatically. +Redpanda Connect provides a simplified xref:develop:connect/connect-quickstart.adoc[quickstart] experience in the UI that helps you to start building data pipelines. The quickstart creates pipelines to stream data into and out of Redpanda using the pipeline editor. === Get Started with Serverless -A Serverless cluster's *Overview* page now provides a *Get Started* wizard to help you start streaming your own data with a xref:develop:connect/connect-quickstart.adoc[Redpanda Connect] pipeline. The wizard lets you stream data into and out of Redpanda without writing producer/consumer code. +A Serverless cluster's *Overview* page now provides a *Get Started* guide to help you start streaming your own data with a xref:develop:connect/connect-quickstart.adoc[Redpanda Connect] pipeline. It lets you stream data into and out of Redpanda without writing producer/consumer code. === Remote read replicas: GA diff --git a/modules/get-started/partials/cloud-quickstart.adoc b/modules/get-started/partials/cloud-quickstart.adoc index 75e6bc38..6d99dbab 100644 --- a/modules/get-started/partials/cloud-quickstart.adoc +++ b/modules/get-started/partials/cloud-quickstart.adoc @@ -1 +1 @@ -TIP: The cluster's *Overview* page includes a *Get Started* wizard to quickly pipeline your data into and out of Redpanda. The wizard guides you to select an input data source (to produce data) or an output data sink (to consume data) and automatically populates the necessary permissions and configurations. See also: xref:develop:connect/connect-quickstart.adoc[Redpanda Connect Quickstart] +TIP: The cluster's *Overview* page includes a *Get Started* guide to help you start streaming data into and out of Redpanda. See also: xref:develop:connect/connect-quickstart.adoc[Redpanda Connect Quickstart] diff --git a/modules/get-started/partials/get-started-serverless.adoc b/modules/get-started/partials/get-started-serverless.adoc index fea7e884..803cfe7e 100644 --- a/modules/get-started/partials/get-started-serverless.adoc +++ b/modules/get-started/partials/get-started-serverless.adoc @@ -9,7 +9,7 @@ When either the credits expire or the days in the trial expire, the clusters mov After you start a trial, Redpanda instantly prepares an account for you. Your account includes a `welcome` cluster with a `hello-world` demo topic you can explore. It includes sample data so you can see how real-time messaging works before sending your own data. -xref:get-started:cluster-types/serverless.adoc#interact-with-your-cluster[Get started] by following the wizard to create a Redpanda Connect glossterm:pipeline[], or by following the steps in the Console to use `rpk` to interact with your cluster from the command line: +xref:get-started:cluster-types/serverless.adoc#interact-with-your-cluster[Get started] by creating a Redpanda Connect glossterm:pipeline[], or by following the steps in the Console to use `rpk` to interact with your cluster from the command line: . Log in with `rpk cloud login`. . Consume from the `hello-world` topic with `rpk topic consume hello-world`.