Introduce tenant provisioning support for ResourceSets

[matheuscscp]

Closes: #388
Closes: #391
Depends on: fluxcd/source-controller#2182
Depends on: fluxcd/pkg#1317

Major new features:

• The opt-in ability to discover, create, reconcile and prune namespaces from .spec.artifacts[].namespace including:
  • /ssa, /reconcile and /prune object-level annotations for namespaces
  • --override-manager and hardcoded kubectl overrides
  • .serviceAccountName and --default-service-account awareness (at least one is required)
  • Drift correction through the AG annotation /reconcileEvery
  • source.extensions.fluxcd.io/externalFinalizer annotation for delegating namespace finalization
• .spec.pathPattern awareness for .spec.artifacts[].namespace
• A new KRM-style configuration API called NamespaceMetadata (file acquired from source artifacts)
• Support for Introduce ExternalArtifact.status.exportedInputs source-controller#2182

Together, these features allow the ultimate tenant provisioning combo. Platform teams will be able to create a monorepo with a well-defined directory structure where tenants request namespaces by creating directories in this well-defined structure. This will be accomplished as follows:

1. Enable Flux multi-tenancy lockdown with --default-service-account (or DEFAULT_SERVICE_ACCOUNT in Flux Operator).
2. Deploy one ArtifactGenerator in the flux-system namespace generating the namespaces from .spec.artifacts[].namespace. Because the AG is the authority for the existence of these namespaces, it has to create them first, then create the ExternalArtifacts inside them after that. And it has to prune them when no more EAs make reference to them (e.g. when the namespace directory is fully deleted, or all subdirectories inside yielding an artifact are deleted). For each EA, the AG will load, parse as map[string]*apiextensionsv1.JSON and copy the contents of .inputsFrom to ExternalArtifact.status.exportedInputs.
3. Deploy one ResourceSetInputProvider type ExternalArtifact in the flux-system namespace selecting the ExternalArtifacts created by the ArtifactGenerator through a label in ArtifactGenerator.spec.commonMetadata in the namespaces selected via flux-operator#1067. The RSIP will export ExternalArtifact.status.exportedInputs for each ExternalArtifact, plus merge the inputs already exported for the ExternalArtifact RSIP type on top (they need priority).
4. Deploy one ResourceSet in the flux-system namespace referencing the RSIP via .spec.inputsFrom. The resource templates can then consume structured inputs coming all the way from the monorepo. These templates can render a Flux Kustomization in the namespace of the ExternalArtifact for applying it, along tenant ServiceAccount and RBAC.

All the three objects above need to set .spec.serviceAccountName to flux-operator (even in ArtifactGenerator, since the source-watcher ServiceAccount will not have RBAC necessary for managing namespaces).

A major use case here for artifact inputs is e.g. defining dependsOn for appliers. In a namespace directory where multiple artifacts are allowed, you will often want to define an order for the appliers of the artifacts.

We are also aligning garbage collection behavior with kustomize-controller, which puts back objects into the inventory and keeps the finalizer when the AG is on deletion in case the API server does not confirm the deletion request for the object. Backoff retries from controller-runtime reattempt the finalization.

---

The API:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
  name: monorepo-tenants
  namespace: flux-system
spec:
  serviceAccountName: flux-operator

  # Drives namespace and artifact discovery.
  pathPattern: "@repo/tenants/{namespace}/modules/{module}"

  # Now gets applied to both ExternalArtifacts and Namespaces.
  commonMetadata:
    labels:
      my.org.io/provisioner: "monorepo"

  sources:
    - alias: repo
      kind: OCIRepository
      name: monorepo

  artifacts:
    - name: "{module}"
      namespace: "{namespace}" # Resolves the pathPattern capture {namespace} to e.g. "some-tenant".
      inputsFrom: "@repo/tenants/{namespace}/modules/{module}/applier-inputs.yaml"
      originRevision: "@repo"
      copy:
        - from: "@repo/tenants/{namespace}/modules/{module}/manifests/**"
          to: "@artifact/"

  # Optional. Allows opting into namespace management by the ArtifactGenerator.
  managedNamespaces:
    # Required. CEL expressions allow-listing namespace names. At least one is required.
    # Receives the namespace name through the 'ns' string variable and must return a boolean.
    allowedNames: [ns.startsWith("tenant-")] # To allow any namespaces use ["true"].

    # Required. Whether or not to delete managed namespaces when they are no longer
    # needed for the artifacts.
    prune: true

    # Optional. The metadata of a desired namespace is constructed as
    # follows:
    #
    #   1. spec.commonMetadata is applied first.
    #   2. Metadata from NamespaceMetadata (with schema defined in
    #      spec.managedNamespaces.metadata.fromSource.{allowedAnnotations,allowedLabels})
    #      is applied next. The NamespaceMetadata API allows sourcing metadata
    #      for a namespace from a file inside a source artifact.
    #   3. Metadata from spec.managedNamespaces.metadata.from is applied
    #      last, potentially overriding previous values. See
    #      spec.managedNamespaces.metadata.from for details.
    metadata:

      # Optional. Sources namespace metadata from source artifacts by reading
      # a NamespaceMetadata file inside the source artifact. This allows, for
      # example, tenants to provide additional metadata for their namespaces
      # for opting into platform features that are enabled via namespace
      # metadata, like observability. The NamespaceMetadata API is defined
      # as follows:
      #
      #   # -----------------------
      #   # namespace-metadata.yaml
      #   # -----------------------
      #   apiVersion: source.extensions.fluxcd.io/v1beta1
      #   kind: NamespaceMetadata
      #   metadata:
      #     annotations:
      #       my.org.io/observability: Enabled
      #     labels:
      #       ...
      fromSource:
        # Required. Path to the file in the source containing the NamespaceMetadata manifest.
        path: "@repo/tenants/{namespace}/namespace-metadata.yaml"

        # The following two fields act as a schema for the metadata, by
        # defining which annotations and labels are allowed and with which
        # values. The value of each annotation or label in these fields
        # must be a CEL expression that returns a boolean value. The
        # expressions receive the value of the annotation or label as
        # the variable 'value'.
        allowedAnnotations:
          my.org.io/observability: "value.matches('^(Enabled|Disabled)$')"
        allowedLabels:
          ...

      # Optional. Authoritative metadata for specific namespaces. Useful,
      # for example, for platform admins to override commonMetadata and
      # metadata sourced from NamespaceMetadata. Applied as a sequence
      # of operations, where each operation is applied on the result of
      # of the previous operation.
      from:
        - # Required. One of Reset, Override, or Merge:
          #
          #   - Reset clears all existing annotations and labels for the specified namespace(s).
          #   - Override merges the specified annotations and labels with the existing ones by
          #     overriding existing values.
          #   - Merge merges only absent annotations and labels, preserving existing values.
          strategy: Override

          # Required. The namespace to which this operation applies, or '*' to indicate
          # all desired namespaces instead of a specific one.
          namespace: my-namespace

          # Optional.
          annotations:
            foo: bar
          # Optional.
          labels:
            baz: qux

status:
  inventory:
    - kind: Namespace
      name: some-tenant
      # Hash of the metadata combined and applied by the AG.
      digest: sha256:589303e73217c92ae0d20d90d425cd143459e2ddc488aa38684d5e69e8c9b4b9
    - kind: ExternalArtifact
      name: some-module
      namespace: some-tenant
      filename: ...
      digest: sha256:dbfd4d986dbccf080479534d20e1c6c6b5893a7378a2dda4038f307e25462b57

[shrinedogg]

This is awesome @matheuscscp! I really like this idea, and I think protecting the namespaces by having to deliberately set the prune tag is a good idea as well. 👍

[matheuscscp]

Ok I think this PR is now well balanced after removing inputsValidation, all APIs come from real world use cases @erikgb helped me figure out, we had a design call yesterday where we reached a point Erik he said he could actually use this in his org.

[erikgb]

Judging by the PR description, this looks like a very useful feature, which could potentially greatly simplify the mechanics in our tenant onboarding repo! 🚀 Thanks, @matheuscscp! One comment:

Pruning is almost always enabled for any GitOps resources in our clusters. We temporarily disable pruning only during migrations. And I don't understand why namespaces (or CRDs) are considered so special. With GitOps, the source of truth is defined in Git, and if Git says that something should cease to exist, it should be deleted by the GitOps engine. 😅 So I would heavily vote for making resource (including namespace) pruning the default. If not, we risk piles of rubbish resources stacking up in our clusters.

[stefanprodan]

One reason not to delete from SW is Kubernetes GC that removes SA + RBAC before anything else, making it impossible for the Flux reconcilers to prune the tenant resources which may expand to cloud services, global resources and resources in other namespaces. The proper way to deal with tenant pruning is by letting the ResourceSet handle it, Flux Operator knows how to wait for Kustomization and HelmRelease finalizers before removing their SA/RBAC.

[matheuscscp]

We can't remove namespace GC entirely here. We need an explicit mechanism to handoff management to something else, otherwise namespaces can leak, and a solution that leaks objects is not GitOps. If the handoff never happened, AG is in charge of GC.

[stefanprodan]

This goes against the current Flux v2 API design, no controller has config files on disk, this was a thing we did in Flux v1 and it proved to be wrong. I would really like to see an RFC motivating this divergence.

[matheuscscp]

First of all, this is not a config file mounted in source-watcher's disk at the application/controller level. It's pulled from inside a source object's artifact.

Second, I strongly disagree with this push back, this small feature does not warrant an RFC. There was no RFC when AG completely changed the way ownership is tracked with source.extensions.fluxcd.io/generator: UID and with how suspension comes from the /reconcile annotation instead of .spec.suspend like in every other Flux CRD. These changed how downstream consumers like Flux CLI and Flux Operator Web UI have to deal with AG in comparison with the older Flux APIs. AG also has its own special inventory format deviating from KS and HR (and RSET and FluxInstance). There are downstream discrepancies for handling AG everywhere already. There was no RFC for any of those discrepancies, and it's not a very involved feature with a very good reason like this one that will require an RFC, especially a feature that does not introduce new discrepancies for downstream consumers like this one. Even flux-schema validation is already able to handle KRM-style config APIs like this, as introduced to support flux-schema's own and flux-mirror's config APIs (which, by the way, are the precedents of this approach in CNCF Flux that you are looking for).

Reason is simple and irrefutable: tenants need the ability to opt into namespace features implemented by the platform team via annotations and labels by specifying them inside the source artifact that is used for discovering the namespace. The whole point of directory-based namespace discovery and management is avoiding the requirement for tenants to create objects that are applied out-of-band. AG has to observe these inputs from a file inside the namespace directory in the monorepo. Even if this was a CRD, and, let's say, AG would be the one applying it, this would be a completely useless behavior. The object would lie in etcd for no reason, serving no purpose.

For flux-schema to be able to validate NamespaceMetadata, we just need to generate and publish its OpenAPI schema like we do for flux-schema itself and flux-mirror. Then flux-schema can consume it the same way.

[stefanprodan]

My concern is that we split the API into CRDs and files on disk which is very uncommon for a controller, the dir where this file is can not be applied by Flux, kubectl, etc. You need to make all these tools ignore it and a .sourceignore will not work, since SW will miss it too.

[matheuscscp]

This controller is very uncommon and innovative. Innovation introduces very uncommon things. Not every innovation can follow every established pattern.

the dir where this file is can not be applied by Flux, kubectl, etc. You need to make all these tools ignore it and a .sourceignore will not work, since SW will miss it too.

This concern is addressed by the directory structure:

  pathPattern: "@repo/tenants/{namespace}/modules/{module}"
  namespaces:
    metadata:
      fromSource:
        path: "@repo/tenants/{namespace}/namespace-metadata.yaml"
  artifacts:
    - name: "{module}"
      namespace: "{namespace}"
      inputsFrom: "@repo/tenants/{namespace}/modules/{module}/applier-inputs.yaml"
      originRevision: "@repo"
      copy:
        - from: "@repo/tenants/{namespace}/modules/{module}/manifests/**"
          to: "@artifact/"

The NamespaceMetadata is not the only file here that cannot be applied by a Kubernetes applier like Flux and kubectl, the applier inputs for processing the artifact (e.g. RSET template inputs) are also a non-Kubernetes YAML file sourced from the monorepo. The directory architechture above (also shown in the PR description) prevents those files from landing in the output artifacts by design.

[stefanprodan]

Did you consider a namespace definition, like we do for copyFrom? The admin could place a namespace YAML in that path, so it serves as the template for all the dinamic ones we create in AG.

[matheuscscp]

Better yet, all the regexes here should be CEL expressions, which also support regex.

[stefanprodan]

Are we safe from tenants creating an AG in their own namespace that would wipe flux-system or cilium? If we never grant SW namespace access, then even if the cluster admin forgets to set --default-account it should still not work.

[matheuscscp]

Better than that, we can require one of .spec.serviceAccountName or --default-service-account to be set for namespace management. With this, even if the controller for some reason had the required RBAC (which Flux users could do even if the upstream distro did not), we never use the controller client and it doesn't work. We force RBAC to be explicit (make impersonation required).

[matheuscscp]

Also, I just noticed Kustomization.spec.prune is actually required, we will do the same here. No more defaults.

[matheuscscp]

Everything since #392 (comment) has been implemented.

Argument is completely invalid, see response.

[matheuscscp]

We can hand over finalization with an annotation in the namespace applied by the external finalizer (the RSET):

source.extensions.fluxcd.io/externalFinalizer: G/V/K/N/N/UID

I chose "finalizer" instead of e.g. "manager" to emphasize that the finalizer has only a single responsibility: deleting the namespace. "Management" still lies with the AG. Finalizer implementations should honor this emphasis (e.g. like the RSET controller will, see below).

When present in a namespace managed by the AG, this annotation will tell AG to check if the finalizer object exists in the cluster, and only if it exists AG will skip GC for the namespace when applicable (i.e. when the number of EAs referencing it reaches zero or the AG is being finalized). The finalizer object reference in the annotation has to be fully qualified, especially with the API Version, otherwise AG has to first look up the kind API versions in the cluster and only then it will be able to look up the object. The UID is also a mandatory part of the reference, because if for some reason the external finalizer has to be recreated and the recreation fails resulting in its new incarnation (with a new UID) never being created or never tracking the namespace in its inventory for GC, then AG is still accountable. In addition, AG will refuse to reconcile the namespace metadata and will error out if this annotation is in its own desired state for the namespace. The annotation has to be guaranteed to come from somewhere else in order not to trick AG to leak the namespace.

Platform team then adds a Namespace manifest in the RSET setting this annotation to the special value self. The RSET controller will recognize the annotation with this special value and will stamp the reference when applying the Namespace manifest, and it will also reject with an error any other metadata for the namespace in its own desired state. This is a deliberate choice from RSET's perspective to emphasize that management of the namespace metadata still lies with the AG no matter what. The namespace metadata features built by this PR into the AG API are what support the intended approach and use cases (notably, allowing tenants to opt into namespace features created by the platform team is a major requirement). Note: this does not mean that AG will not play nice with non-kubectl field managers, so other controllers or kubectl --field-manager=... are still allowed. Override manager features will stay, namespace-level annotatios /ssa, /reconcile and /prune will stay.

When all the EAs in the namespace vanish, the namespace will no longer appear in the RSET rendered manifests, RSET controller will notice and perform the ordered GC.

[matheuscscp]

@stefanprodan AI converged after 5~6 review-fix cycles.