feat: Support 8.18.0/9.0.0 and EDOT (#1081)

* feat: demo EDOT

* test: update docker configuration

* test: ignore tests

* fix: update index patterns

* fix: links

* fix: retrieve logs

* chore: remove deprecated

* fix: support APM and EDOT

* fix: add demo command and docs

* fix: migrate to apache-httpcomponents-client-5-api
fix: review ependencies

* fix: update test

* test: junit5

* test: stop SDK

* fix: HttpContext cast

* test: ignore IT test for the moment

* test: use @disabled from JUnit5

* fix: select transaction type

* feat: grafana demo

* Apply suggestions from code review

Co-authored-by: Victor Martinez <[email protected]>

---------

Co-authored-by: Victor Martinez <[email protected]>

Author: kuisathaverat

.editorconfig

@@ -31,3 +31,7 @@ insert_final_newline = true
 [*.java]
 indent_style = space
 trim_trailing_whitespace = true
+
+[Makefile]
+indent_style = tab
+trim_trailing_whitespace = true

README.md

@@ -13,19 +13,21 @@ Monitor and observe Jenkins with OpenTelemetry.
 
 Visualize jobs and pipelines executions as distributed traces:
 
-<img alt="SpringBootPipeline Execution Trace" width="400px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-maven-execution-trace-jaeger.png" />
-<p style="font-size:0.75em; font-style: italic">Example pipeline execution trace of a SpringBoot Maven going application</p>
+![SpringBootPipeline Execution Trace](docs/images/jenkins-maven-execution-trace-jaeger.png)
 
+*Example pipeline execution trace of a SpringBoot Maven going application*
 
 Visualize Jenkins and pipeline health indicators:
 
-<img alt="Example Kibana dashboard of the Jenkins and CI jobs health" width="400px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/kibana_jenkins_overview_dashboard.png" />
-<p style="font-size:0.75em; font-style: italic">Example Kibana dashboard of the Jenkins and CI jobs health</p>
+![Example Kibana dashboard of the Jenkins and CI jobs health](docs/images/kibana_jenkins_overview_dashboard.png)
+
+*Example Kibana dashboard of the Jenkins and CI jobs health*
 
 Troubleshoot Jenkins performances with distributed tracing of HTTPs requests:
 
-<img alt="Jenkins HTTP request trace with Jaeger" width="400px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/http-tracing/jenkins-http-request-trace-jaeger.png" />
-<p style="font-size:0.75em; font-style: italic">Example Jenkins HTTP trace</p>
+![Jenkins HTTP request trace with Jaeger](docs/images/http-tracing/jenkins-http-request-trace-jaeger.png)
+
+*Example Jenkins HTTP trace*
 
 ## Architecture
 
@@ -37,7 +39,7 @@ Here are example architectures with Elastic, Jaeger, and Prometheus:
 
 | CI/CD Observability with Jaeger and Prometheus                                                                                                                                                                            | CI/CD Observability with Elastic                                                                                                                                                                                |
 |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| <img alt="Jenkins monitoring with Jaeger and Prometheus" width="400" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-opentelemetry-architecture-jaeger-prometheus.png" > | <img alt="Jenkins monitoring with Elastic Observability" width="400" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-opentelemetry-architecture-elastic.png" > |
+| ![Jenkins monitoring with Jaeger and Prometheus](docs/images/jenkins-opentelemetry-architecture-jaeger-prometheus.png) | ![Jenkins monitoring with Elastic Observability](docs/images/jenkins-opentelemetry-architecture-elastic.png) |
 
 ## Getting started
 
@@ -63,9 +65,9 @@ Here are example architectures with Elastic, Jaeger, and Prometheus:
 * Set up Jenkins health dashboards on your OpenTelemetry metrics visualization solution. See details including guidance
   for Elastic Kibana [here](https://github.com/jenkinsci/opentelemetry-plugin/blob/master/docs/monitoring-metrics.md).
 
-<img alt="Sample Configuration" width="300px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-opentelemetry-plugin-configuration.png" />
+![Sample Configuration](docs/images/jenkins-opentelemetry-plugin-configuration.png)
 
-<p style="font-size:0.75em; font-style: italic">Example Jenkins OpenTelemetry configuration</p>
+*Example Jenkins OpenTelemetry configuration*
 
 ## Setup and Configuration
 
@@ -78,33 +80,29 @@ see [Setup and Configuration](https://github.com/jenkinsci/opentelemetry-plugin/
 For details on how to explore and troubleshoot jobs and pipelines builds as traces,
 see [Traces of Jobs and Pipeline Builds](https://github.com/jenkinsci/opentelemetry-plugin/blob/master/docs/job-traces.md).
 
-<img alt="SpringBootPipeline Execution Trace" width="300px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-maven-execution-trace-jaeger.png" />
-<p style="font-size:0.75em; font-style: italic">Example pipeline execution trace of a SpringBoot app built with Maven going through security checks with Snyk, deployed on a Maven repository and published as a Docker image</p>
+![SpringBootPipeline Execution Trace](docs/images/jenkins-maven-execution-trace-jaeger.png)
+
+*Example pipeline execution trace of a SpringBoot app built with Maven going through security checks with Snyk, deployed on a Maven repository and published as a Docker image*
 
 ## Troubleshooting pipeline plugins and the execution on the Jenkins build agents
 
 For details on the execution of pipeline plugin steps on the Jenkins build agents,
 activate tracing in the Jenkins build agents using:
 
-```
+```properties
 otel.instrumentation.jenkins.agent.enabled=true
 ```
 
 To activate detailed traces of the communication from the Jenkins Controller to the Jenkins Agents, activate the
 instrumentation of Jenkins remoting with:
 
-```
+```properties
 otel.instrumentation.jenkins.remoting.enabled=true
 ```
 
 Note that the instrumentation of Jenkins remoting is not feature complete and may not capture all the communication
 between the Jenkins Controller and the Jenkins Agents.
 
-<!--
-TODO add once image is merged is the `main` branch
-<img alt="Jenkins Remoting instrumentation" width="300px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-remoting-instrumentation.png" />
--->
-
 ## Troubleshooting Jenkins Performances Using Tracing on the HTTP Requests of the Jenkins Controller
 
 For details to set up Jenkins with Elastic, Jaeger or Prometheus, to configure the integration including using Jenkins
@@ -123,8 +121,9 @@ attributes [here](https://github.com/jenkinsci/opentelemetry-plugin/blob/master/
 For details on how to store Jenkins pipelines build logs in an Observability backend like Elastic or Loki,
 see [Storing Jenkins Pipeline Logs in an Observability Backend though OpenTelemetry](https://github.com/jenkinsci/opentelemetry-plugin/blob/master/docs/build-logs.md).
 
-<img alt="Storing Jenkins pipeline logs in Elasticsearch and visualizing logs both in Kibana and through Jenkins GUI" width="300px" src="https://raw.githubusercontent.com/jenkinsci/opentelemetry-plugin/master/docs/images/jenkins-pipeline-logs-elastic-with-visualization-through-jenkins.png" />
-<p style="font-size:0.75em; font-style: italic">Storing Jenkins pipeline logs in Elasticsearch and visualizing logs both in Kibana and through Jenkins GUI</p>
+![Storing Jenkins pipeline logs in Elasticsearch and visualizing logs both in Kibana and through Jenkins GUI](docs/images/jenkins-pipeline-logs-elastic-with-visualization-through-jenkins.png)
+
+*Storing Jenkins pipeline logs in Elasticsearch and visualizing logs both in Kibana and through Jenkins GUI*
 
 ## Other CI/CD Tools supporting OpenTelemetry traces
 
@@ -182,7 +181,7 @@ node {
 Navigate to the Jenkins OpenTelemetry Plugin configuration, in the "Advanced" section, add to the "Configuration
 Properties text area the following:
 
-```
+```properties
 otel.exporter.otlp.protocol=http/protobuf
 ```
 
@@ -192,17 +191,17 @@ No test have been done on disabling the Groovy Sandbox and accessing the Jenkins
 Jenkins OpenTelemetry Plugin for the following reasons:
 
 * Disabling the Groovy Sandbox is a very advanced use case due to the security implications of doing so
-* The surface of Jenkins pipeline logs capabilities exposed by disabling the Groovy sandbox is very broad and goes way
-  beyond the OpenTelemetyr plugin
+* The surface of Jenkins pipeline logs capabilities exposed by disabling the Groovy sandbox is very broad and goes way beyond the OpenTelemetry plugin
 
 If you are limited with the current capabilities of the Jenkins OpenTelemetry Plugin and consider opening up the Groovy
-sandbox to workaround these limitations, please prefer to reach out to us creating an enhancement request so we can work
-together at productizing the proper secured solution to your problem.
+sandbox to workaround these limitations, please prefer to reach out to us creating an enhancement request so we can work together at productizing the proper secured solution to your problem.
 
 ## Learn More
 
 * You can look at this video tutorial to get
-  started: [![Tracing Your Jenkins Pipelines With OpenTelemetry and Jaeger](https://img.youtube.com/vi/3XzVOxvNpGM/0.jpg)](https://www.youtube.com/watch?v=3XzVOxvNpGM)
+  started:
+  
+  [![Tracing Your Jenkins Pipelines With OpenTelemetry and Jaeger](https://img.youtube.com/vi/3XzVOxvNpGM/0.jpg)](https://www.youtube.com/watch?v=3XzVOxvNpGM)
 * [DevOpsWorld 2021 - Embracing Observability in Jenkins with OpenTelemetry](https://www.devopsworld.com/agenda/session/581459)
 
 ## Demos
@@ -216,5 +215,3 @@ Refer to our [contribution guidelines](https://github.com/jenkinsci/.github/blob
 ## LICENSE
 
 Licensed under Apache Software License 2, see [LICENSE](LICENSE)
-
-

demos/Makefile

@@ -1,20 +1,29 @@
 .DEFAULT_GOAL := start
 
 export OTELCOL_IMG?=otel/opentelemetry-collector-contrib-dev:latest
-export ELASTIC_STACK_VERSION?=8.18.0
+export ELASTIC_STACK_VERSION?=9.0.0
 
 .PHONY: build
 build:
-	@env docker compose --file docker-compose.yml build
+	@env docker compose --file docker-compose.yml build --no-cache
 
 .PHONY: start
 start:
 	@env docker compose --file docker-compose.yml up --detach --no-recreate
 
+.PHONY: start-edot
+start-edot: OTEL_EXPORTER_OTLP_ENDPOINT?=http://edot:4317
+start-edot:
+	@env docker compose --file docker-compose.yml up --detach --no-recreate jenkins jenkins-inbound-agent edot elasticsearch kibana
+
+.PHONY: start-grafana
+start-grafana:
+	@env docker compose --file docker-compose-grafana.yml up --detach --no-recreate
+
 .PHONY: stop
 stop:
-	@env docker compose --file docker-compose.yml stop
+	@env docker compose --file docker-compose.yml --file docker-compose-grafana.yml stop
 
 .PHONY: clean
 clean:
-	@env docker compose --file docker-compose.yml down -v
+	@env docker compose --file docker-compose.yml --file docker-compose-grafana.yml down -v

demos/config/grafana-datasources.yml

@@ -0,0 +1,16 @@
+apiVersion: 1
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    isDefault: true
+  - name: prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+  - name: Jaeger
+    type: jaeger
+    access: proxy
+    url: http://jaeger-all-in-one:16686
+

demos/config/hello-world.dsl

@@ -0,0 +1,19 @@
+NAME = 'hello_world'
+DSL = """pipeline {
+  agent any
+  stages {
+    stage('hello') {
+      steps {
+        echo 'Hello, world!'
+      }
+    }
+  }
+}"""
+
+pipelineJob(NAME) {
+  definition {
+    cps {
+      script(DSL.stripIndent())
+    }
+  }
+}

demos/config/jenkins-elastic.yml

@@ -0,0 +1,61 @@
+jenkins:
+  systemMessage: "Configured automatically with JCasC plugin"
+  crumbIssuer:
+    standard:
+      excludeClientIPFromCrumb: false
+  labelString: flyweight
+  mode: EXCLUSIVE
+  numExecutors: 5
+  remotingSecurity:
+    enabled: false
+  scmCheckoutRetryCount: 2
+  nodes:
+  - permanent:
+      labelString: "local"
+      launcher:
+        inbound:
+          workDirSettings:
+            disabled: true
+            failIfWorkDirIsMissing: false
+            internalDir: "remoting"
+            workDirPath: "/tmp"
+      name: "local"
+      numExecutors: 2
+      remoteFS: "/tmp"
+      retentionStrategy: "always"
+credentials:
+  system:
+    domainCredentials:
+    - credentials:
+      - usernamePassword:
+          description: "elasticsearch-logs"
+          id: "elasticsearch-logs"
+          password: "${ELASTICSEARCH_PASSWORD}"
+          scope: GLOBAL
+          username: "${ELASTICSEARCH_USERNAME}"
+unclassified:
+  location:
+    url: ${JENKINS_URL}
+  openTelemetry:
+    authentication: "noAuthentication"
+    endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
+    exportOtelConfigurationAsEnvironmentVariables: true  # Note that OpenTelemetry credentials, if configured, will be exposed as environment variables
+    observabilityBackends:
+      - elastic:
+          displayKibanaDashboardLink: true
+          elasticLogsBackend:
+            elasticLogsBackendWithJenkinsVisualization:
+              elasticsearchCredentialsId: "elasticsearch-logs"
+              elasticsearchUrl: ${ELASTICSEARCH_HOST}
+          kibanaBaseUrl: ${KIBANA_HOST}
+          name: "Elastic Observability"
+          enableEDOT: true
+jobs:
+  - file: "/var/jenkins_home/config/ansible.dsl"
+  - file: "/var/jenkins_home/config/gradle.dsl"
+  - file: "/var/jenkins_home/config/maven.dsl"
+  - file: "/var/jenkins_home/config/otel-cli.dsl"
+  - file: "/var/jenkins_home/config/otel-cli-make.dsl"
+  - file: "/var/jenkins_home/config/import_dashboard.dsl"
+  - file: "/var/jenkins_home/config/pytest-otel.dsl"
+  - file: "/var/jenkins_home/config/hello-world.dsl"

demos/config/jenkins.yml renamed to demos/config/jenkins-grafana.yml

@@ -30,25 +30,22 @@ credentials:
       - usernamePassword:
           description: "elasticsearch-logs"
           id: "elasticsearch-logs"
-          password: "changeme"
+          password: "${ELASTICSEARCH_PASSWORD}"
           scope: GLOBAL
-          username: "jenkins_logs_user"
+          username: "${ELASTICSEARCH_USERNAME}"
 unclassified:
   location:
-    url: "http://jenkins:8080/"
+    url: ${JENKINS_URL}
   openTelemetry:
     authentication: "noAuthentication"
-    endpoint: "http://otel-collector:4317"
+    endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
     exportOtelConfigurationAsEnvironmentVariables: true  # Note that OpenTelemetry credentials, if configured, will be exposed as environment variables
     observabilityBackends:
-    - elastic:
-        displayKibanaDashboardLink: true
-        elasticLogsBackend:
-          elasticLogsBackendWithJenkinsVisualization:
-            elasticsearchCredentialsId: "elasticsearch-logs"
-            elasticsearchUrl: "http://elasticsearch:9200"
-        kibanaBaseUrl: "http://localhost:5601"
-        name: "Elastic Observability"
+      - grafana:
+          grafanaBaseUrl: ${GRAFANA_HOST}
+          grafanaLogsBackend:
+            grafanaLogsBackendWithJenkinsVisualization:
+              lokiUrl: ${LOKI_HOST}
 jobs:
   - file: "/var/jenkins_home/config/ansible.dsl"
   - file: "/var/jenkins_home/config/gradle.dsl"
@@ -57,3 +54,4 @@ jobs:
   - file: "/var/jenkins_home/config/otel-cli-make.dsl"
   - file: "/var/jenkins_home/config/import_dashboard.dsl"
   - file: "/var/jenkins_home/config/pytest-otel.dsl"
+  - file: "/var/jenkins_home/config/hello-world.dsl"

demos/config/otel-collector-config.yaml

@@ -3,9 +3,8 @@ receivers:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317
-      http: 
+      http:
         endpoint: 0.0.0.0:4318
-
 exporters:
   prometheus:
     endpoint: "0.0.0.0:8889"

demos/config/otel-collector-grafana.yaml

@@ -0,0 +1,51 @@
+---
+extensions:
+  health_check:
+  pprof:
+    endpoint: :1888
+  zpages:
+    endpoint: :55679
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+exporters:
+  prometheus:
+    endpoint: "0.0.0.0:8889"
+    namespace: promexample
+    resource_to_telemetry_conversion:
+      enabled: true
+    const_labels:
+      label1: value1
+  debug:
+    verbosity: detailed
+    sampling_initial: 5
+    sampling_thereafter: 200
+  otlphttp:
+    endpoint: http://loki:3100/otlp
+  otlp/jaeger:
+    endpoint: jaeger-all-in-one:4317
+    tls:
+      insecure: true
+processors:
+  batch:
+
+service:
+  extensions: [pprof, zpages, health_check]
+  pipelines:
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlphttp]
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlp/jaeger]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, prometheus]

demos/config/plugins.txt

@@ -6,7 +6,7 @@ git
 filesystem_scm
 jdk-tool
 job-dsl
-opentelemetry
+opentelemetry::https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/opentelemetry/3.1527.vd26606274117/opentelemetry-3.1527.vd26606274117.hpi
 pipeline-model-definition
 swarm
 workflow-aggregator