Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ All notable changes to this project will be documented in this file.
- Set `nifi.content.repository.archive.max.retention.period` to `3 days` (previously empty, which NiFi interprets as `Long.MAX_VALUE` and effectively disables time-based archive purge). Without a time-based ceiling, the content archive can grow to half the content PVC and accumulate millions of files, which makes the synchronous startup directory scan in `FileSystemRepository.initializeRepository` very slow. Users requiring a longer content-replay window can extend via `configOverrides`. The provenance audit trail is independent of this setting and unaffected ([#936]).
- test: Bump vector-aggregator to 0.55.0, replace /graphql call with gRPC call ([#940]).

### Removed

- BREAKING: Remove support for NiFi 1.x.
This removes the Prometheus reporting-task Job (and its `spec.clusterConfig.createReportingTaskJob` field), the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x.
`status.deployed_version` is retained even though it no longer drives the (now removed) non-rolling upgrade state machine, as we don't want a breaking change to the status just for this ([#954]).

### Fixed

- Fix broken link to the NiFi authorization usage guide in the `spec.clusterConfig.authorization` CRD doc (`usage-guide` -> `usage_guide`) ([#924]).
Expand All @@ -34,6 +40,7 @@ All notable changes to this project will be documented in this file.
[#935]: https://github.com/stackabletech/nifi-operator/pull/935
[#936]: https://github.com/stackabletech/nifi-operator/pull/936
[#940]: https://github.com/stackabletech/nifi-operator/pull/940
[#954]: https://github.com/stackabletech/nifi-operator/pull/954

## [26.3.0] - 2026-03-16

Expand Down
11 changes: 0 additions & 11 deletions deploy/helm/nifi-operator/templates/clusterrole-operator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,17 +81,6 @@ rules:
- list
- patch
- watch
# Optional reporting-task Job (NiFi 1.x only). Applied via SSA and tracked for orphan cleanup.
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
# PodDisruptionBudget created per role. Applied via SSA and tracked for orphan cleanup.
- apiGroups:
- policy
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Every role group is accessible through it's own Service, and there is a Service

== Dependencies

Apache NiFi 1.x depends on Apache ZooKeeper which you can run in Kubernetes with the xref:zookeeper:index.adoc[].
NiFi builds its cluster quorum using Kubernetes by default and needs no additional dependencies.
It can optionally use Apache ZooKeeper instead, which you can run in Kubernetes with the xref:zookeeper:index.adoc[].

== [[demos]]Demos

Expand Down
2 changes: 0 additions & 2 deletions docs/modules/nifi/pages/usage_guide/clustering.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ CAUTION: The cluster backend of an existing cluster should never be changed. Oth
[#backend-kubernetes]
== Kubernetes

NOTE: The Kubernetes provider is only supported by Apache NiFi 2.0 or newer. When using NiFi 1.x, use the xref:#backend-zookeeper[] backend instead.

The Kubernetes backend is used by default (unless the xref:#backend-zookeeper[] backend is configured), and stores all state in Kubernetes objects, in the same namespace as the `NifiCluster` object.

It takes no configuration.
Expand Down
33 changes: 1 addition & 32 deletions docs/modules/nifi/pages/usage_guide/monitoring.adoc
Original file line number Diff line number Diff line change
@@ -1,39 +1,8 @@
= Monitoring
:description: The Stackable Operator for Apache NiFi automatically configures NiFi to export Prometheus metrics.
:k8s-job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
:k8s-network-policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
:prometheus-operator: https://prometheus-operator.dev/

In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`,window=_blank].

The NiFi `2.0.0` release changed the way of exposing Prometheus metrics significantly.
The following steps explain on how to expose Metrics in NiFi versions `1.x.x` and `2.x.x`.

== Configure metrics in NiFi `1.x.x`

For NiFi versions `1.x.x`, the operator automatically configures NiFi to export Prometheus metrics.
This is done by creating a {k8s-job}[Job,window=_blank] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task,window=_blank].

IMPORTANT: Network access from the Job to NiFi is required.
If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies,window=_blank], make sure to allow access from the Job to NiFi.

See xref:operators:monitoring.adoc[] for more details.

== Disabling create-reporting-task Job

It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.

To achieve this use the following configuration:

[source,yaml]
----
spec:
clusterConfig:
createReportingTaskJob:
enabled: false
----

== Configure metrics in NiFi `2.x.x`
== Configure metrics

The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank].
Metrics are now always exposed and can be scraped using the NiFi `metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`.
Expand Down
19 changes: 0 additions & 19 deletions docs/modules/nifi/pages/usage_guide/overrides.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,25 +96,6 @@ Pod overrides allow you to configure any attributes that can be configured on a

Read the xref:concepts:overrides.adoc#pod-overrides[Pod overrides concepts page] to learn more.

=== Pod overrides on create-reporting-task Job

In addition to podOverrides on the created StatefulSet we also support podOverrides on the created Kubernetes Job, which enables the export of Prometheus metrics within NiFi.

[source,yaml]
----
spec:
clusterConfig:
createReportingTaskJob:
# enabled: false # You can also turn off the Job entirely
podOverrides: # podOverrides as usual
spec:
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
----

== JVM argument overrides

Stackable operators automatically determine the set of needed JVM arguments, such as memory settings or trust- and keystores.
Expand Down
14 changes: 1 addition & 13 deletions docs/modules/nifi/pages/usage_guide/updating.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,4 @@ spec:

<1> Change the NiFi version here

[WARNING]
====
NiFi clusters cannot be upgraded or downgraded in a rolling fashion due to a limitation in NiFi prior to version 2.

When upgrading between NiFi 1 versions or from NiFi 1 to NiFi 2, any change to the NiFi version in the CRD triggers a full cluster restart with brief downtime.
However, the Stackable image version can be updated in a rolling manner, provided the NiFi version remains unchanged.

Since NiFi version 2, rolling upgrades are supported.
====

== NiFi 2.0.0

Before you can upgrade to `2.0.0` you https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance[need to update] to at least version 1.27.x!
Rolling upgrades are supported, so changing the NiFi version in the CRD updates the cluster without downtime.
45 changes: 1 addition & 44 deletions extra/crds.yaml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is ok to just remove. I can't think of another way, and the config is no longer supported, so I guess it is fine.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I was also thinking of using CRD versioning, but IIRC we only support renames so far

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could deprecate it, but are we then keeping it forever? We will discuss in the team with @Techassi as well

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we do need a strategy for dealing with removals.

deprecating is also a breaking change for the user (unless it is done in a new version, but then we might as well do the breaking removal there).

Original file line number Diff line number Diff line change
Expand Up @@ -139,33 +139,6 @@ spec:
- accessPolicyProvider
type: object
type: object
createReportingTaskJob:
default:
enabled: true
podOverrides: {}
description: |-
This section creates a `create-reporting-task` Kubernetes Job, which enables the export of
Prometheus metrics within NiFi.
properties:
enabled:
default: true
description: |-
Whether the Kubernetes Job should be created, defaults to true. It can be helpful to disable
the Job, e.g. when you configOverride an authentication mechanism, which the Job currently
can't use to authenticate against NiFi.
type: boolean
podOverrides:
default: {}
description: |-
Here you can define a
[PodTemplateSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#podtemplatespec-v1-core)
to override any property that can be set on the Pod of the create-reporting-task Kubernetes Job.
Read the
[Pod overrides documentation](https://docs.stackable.tech/home/nightly/concepts/overrides#pod-overrides)
for more information.
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
customComponentsGitSync:
default: []
description: |-
Expand Down Expand Up @@ -347,26 +320,11 @@ spec:
`nifiPbkdf2AesGcm256` (the default value),
`nifiArgon2AesGcm256`,

The following algorithms are deprecated and will be removed in future versions:

`nifiArgon2AesGcm128`,
`nifiBcryptAesGcm128`,
`nifiBcryptAesGcm256`,
`nifiPbkdf2AesGcm128`,
`nifiScryptAesGcm128`,
`nifiScryptAesGcm256`.

Learn more about the specifics of the algorithm parameters in the
[NiFi documentation](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms).
enum:
- nifiPbkdf2AesGcm256
- nifiArgon2AesGcm256
- nifiBcryptAesGcm128
- nifiBcryptAesGcm256
- nifiPbkdf2AesGcm128
- nifiArgon2AesGcm128
- nifiScryptAesGcm128
- nifiScryptAesGcm256
- null
nullable: true
type: string
Expand Down Expand Up @@ -413,8 +371,7 @@ spec:
When using the [Stackable operator for Apache ZooKeeper](https://docs.stackable.tech/home/nightly/zookeeper/)
to deploy a ZooKeeper cluster, this will simply be the name of your ZookeeperCluster resource.

The Kubernetes provider will be used if this field is unset. Kubernetes is only supported for NiFi 2.x and newer,
NiFi 1.x requires ZooKeeper.
The Kubernetes provider will be used if this field is unset.
type: string
required:
- authentication
Expand Down
49 changes: 5 additions & 44 deletions rust/operator-binary/src/config/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use std::{

use jvm::build_merged_jvm_config;
use product_config::{ProductConfigManager, types::PropertyNameKind};
use snafu::{ResultExt, Snafu, ensure};
use snafu::{ResultExt, Snafu};
use stackable_operator::{
commons::resources::Resources,
crd::git_sync,
Expand All @@ -20,7 +20,6 @@ use strum::{Display, EnumIter};
use crate::{
crd::{
HTTPS_PORT, NifiConfig, NifiRole, NifiRoleType, NifiStorageConfig, PROTOCOL_PORT,
sensitive_properties,
v1alpha1::{self, NifiClusteringBackend},
},
operations::graceful_shutdown::graceful_shutdown_config_properties,
Expand Down Expand Up @@ -101,14 +100,6 @@ pub enum Error {

#[snafu(display("failed to generate OIDC config"))]
GenerateOidcConfig { source: oidc::Error },

#[snafu(display(
"NiFi 1.x requires ZooKeeper (hint: upgrade to NiFi 2.x or set .spec.clusterConfig.zookeeperConfigMapName)"
))]
Nifi1RequiresZookeeper,

#[snafu(display("failed to configure sensitive properties"))]
ConfigureSensitiveProperties { source: sensitive_properties::Error },
}

/// Create the NiFi bootstrap.conf
Expand Down Expand Up @@ -156,35 +147,14 @@ pub fn build_nifi_properties(
proxy_hosts: &str,
auth_config: &NifiAuthenticationConfig,
overrides: BTreeMap<String, String>,
product_version: &str,
git_sync_resources: &git_sync::v1alpha2::GitSyncResources,
) -> Result<String, Error> {
// TODO: Remove once we dropped support for all NiFi 1.x versions
let is_nifi_1 = product_version.starts_with("1.");

let mut properties = BTreeMap::new();
// Core Properties
// According to https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance#MigrationGuidance-Migratingto2.0.0-M1
// The nifi.flow.configuration.file property in nifi.properties must be changed to reference
// "flow.json.gz" instead of "flow.xml.gz"
// TODO: Remove once we dropped support for all 1.x.x versions
// TODO(malte): In order to use CLI tools like: ./bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
// we have to set both "nifi.flow.configuration.file" and "nifi.flow.configuration.json.file" in NiFi 1.x.x.
if is_nifi_1 {
properties.insert(
"nifi.flow.configuration.file".to_string(),
NifiRepository::Database.mount_path() + "/flow.xml.gz",
);
properties.insert(
"nifi.flow.configuration.json.file".to_string(),
NifiRepository::Database.mount_path() + "/flow.json.gz",
);
} else {
properties.insert(
"nifi.flow.configuration.file".to_string(),
NifiRepository::Database.mount_path() + "/flow.json.gz",
);
}
properties.insert(
"nifi.flow.configuration.file".to_string(),
NifiRepository::Database.mount_path() + "/flow.json.gz",
);

properties.insert(
"nifi.flow.configuration.archive.enabled".to_string(),
Expand Down Expand Up @@ -535,10 +505,6 @@ pub fn build_nifi_properties(
.clone()
.unwrap_or_default();

sensitive_properties_algorithm
.check_for_nifi_version(spec.image.product_version())
.context(ConfigureSensitivePropertiesSnafu)?;

properties.insert(
"nifi.sensitive.props.algorithm".to_string(),
sensitive_properties_algorithm.to_string(),
Expand Down Expand Up @@ -635,8 +601,6 @@ pub fn build_nifi_properties(
}

v1alpha1::NifiClusteringBackend::Kubernetes {} => {
ensure!(!is_nifi_1, Nifi1RequiresZookeeperSnafu);

properties.insert(
"nifi.cluster.leader.election.implementation".to_string(),
"KubernetesLeaderElectionManager".to_string(),
Expand All @@ -653,9 +617,6 @@ pub fn build_nifi_properties(
//####################
// Custom components #
//####################
// NiFi 1.x does not support Python components and the Python configuration below is just
// ignored.

// The command used to launch Python.
// This property must be set to enable Python-based processors.
properties.insert("nifi.python.command".to_string(), "python3".to_string());
Expand Down
Loading
Loading