ApplicationSet and secret paths

[huehnerhose]

Hi,

I was not able to find how to solve this, though I have the constant feeling of not asking the right search prompts.

I use app-of-apps approach to deploy ApplicationSets, deploying e.g. qa and prod env per Application. So I have an ApplicationSet "apps-of-apps" deploying via list-generator multiple Application, e.g. Application: "supporting-services" which deployes ApplicationSet "frontend-service", which deploys "frontend-service-prod" and "frontend-service-qa". The applications themselves are helm deployments via source.helm.values. So the applicationSet (reduced) looks like:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: frontend-service
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - project: "qa"
        namespace: qa
        vaultPath: "dev"
      - project: "prod"
        namespace: prod
        vaultPath: "prod"

  template:
    metadata:
      name: frontend-service-{{.project}}
    spec:
      destination:
        namespace: '{{.namespace}}'
        server: 'https://kubernetes.default.svc'
      project: '{{.project}}'
      source:
        repoURL: helm.repo.com
        targetRevision: '3.2.12'
        chart: frontend-service
        plugin:
          env:
            - name: TEST
              value: FOO
        helm:
          values: |
            env:
              API_KEY: "<path:myapplication-{{.vaultPath}}-secrets/data/mySecret#API_KEY>"

You probably already see the issue.

API_KEY: "<path:myapplication-{{.vaultPath}}-secrets/data/mySecret#API_KEY>"

breaks the argocd-vault-plugin generate run, since the {{.vaultPath}} replacement / templating happens after the AVP is running. So AVP gets the literal string <path:myapplication-{{.vaultPath}}-secrets/data/mySecret#API_KEY> as path to look for the secret.

I tried, as you can see with the source.plugin.env part to pass ENV to the CMP/AVP run to run some replacement prior to the actual run of argocd-vault-plugin generate. But I never get a run on this ApplicationSet / resulting Application. But always only on the Application "supporting-services".

I moved the avp-annotation between the different Application Manifests, without any result.

Long Story short: I am lost if my setup is stupid to begin with and / or can work with AVP at all?

Edit:
I deploy AVP as sidecar in the argo-cd helm chart via

          cmp:
            create: true
            plugins:
              avp:
                allowConcurrency: true
                discover:
                  find:
                    command:
                      - sh
                      - "-c"
                      - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
                generate:
                  command:
                    - sh
                    - "-c"
                    - "argocd-vault-plugin generate -s \"argocd:vault-configuration\" ."

[pre]

We have solved this issue with our own ArgoCD Config Management Plugin which pre-processes the manifests before passing them to argocd-vault-plugin.

[Tazer]

We have solved this issue with our own ArgoCD Config Management Plugin which pre-processes the manifests before passing them to argocd-vault-plugin.

Anything that is open source? We also have needs for this use case ☺️

[pre]

Anything that is open source?

Unfortunately not, but the whole is quite straightforward to do. You can get started quickly with a plugin that just passes through all the manifests as is. Once you have hold on how the ArgoCD CMP thing is working, it's not difficult to add an envsubst on top of it.

[parkjeongryul]

Hi @pre,

I'm confused about when your custom CMP runs.

The generator values ({{.vaultPath}}) are substituted when ApplicationSet creates Applications. But AVP needs these values to be already substituted.

Does your custom CMP preprocess by actually generating the Applications from ApplicationSet first, before running AVP?

Thanks!

[pre]

Hi!

We write a Config Management Plugin like this:

apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: kustomize-ourenvsubst-vault
spec:
  generate:
    command: ["bash", "-cEeuo", "pipefail"]
    args:
    - |
      kustomize build --load-restrictor LoadRestrictionsNone . \
        | our-envsubst \
        | vault-with-project-approle generate -

Here our-envsubst is the command which performs the env variable substitution.
And vault-with-project-approle is a wrapper shell script which performs some initial setup (decide Vault AppRole name depending on the Application) before yielding to argocd-vault-plugin.

Both Application and ApplicationSet resources can use this CMP with spec.source.plugin.name: kustomize-ourenvsubst-vault. We don't use the discovery mechanism, since all our manifests are deterministic and the explicit approach works better. In other words, for us it's known beforehand which CMP is needed for a given Application or ApplicationSet.

We have also other CMPs. For example kustomize-ourenvsubst performs only the env variable substitution without replacing any Vault placeholders. In practice, that is exactly the same ConfigManagementPlugin manifest but without the last pipe (see above).

[pre]

In other words, find out what's needed to generate the manifests in an interactive shell. Then wrap those commands into bash scripts which you invoke from a ConfigManagementPlugin.

[parkjeongryul]

Thanks for the explanation!
If I understand correctly, you're saying we should disable discovery and explicitly specify the plugin to make it work.

But with this approach, due to this issue (argoproj/argo-cd#13413), it seems like we'd have to give up on ArgoCD's native Helm plugin.

If we abandon the native Helm plugin, won't that make maintenance more difficult going forward? Or would this not matter much?