docs(proposals): add KEP-3165 for filesystem webhook certificates

[shubhM13]

Purpose of this PR

This adds the provisional design document for #3165. The Spark Operator webhook
currently obtains serving certificates through its self-signed or cert-manager
paths, both of which assume certificate material is stored in a Kubernetes
Secret and copied into the serving directory. Clusters whose platform PKI
delivers short-lived certificates directly as files have no supported path that
preserves live rotation and avoids unnecessary Secret ownership.

Proposed changes:

• Define an opt-in filesystem certificate provider while preserving the
  existing self-signed default and cert-manager compatibility flag.
• Separate serving-certificate source from admission caBundle ownership, with
  exactly one operator or external writer.
• Specify validated hot reload, CA overlap, readiness, migration, rollback,
  least-privilege RBAC, observability, and test requirements.
• Record the controller-runtime watcher behavior that needs maintainer direction
  before implementation.

Fixes #3165

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Admission webhooks are fail-closed, and serving-certificate and trust rotation
span filesystem generations, two admission objects, API-server propagation, and
multiple webhook replicas. Agreeing on the ownership and rotation contract
before implementation avoids encoding an unsafe or incompatible API.

The two points where maintainer direction is most useful are in
Open Questions:

1. Whether to enhance controller-runtime's certificate watcher or approve a
   small bounded polling loader in Spark Operator.
2. Whether the initial chart should support generic injector annotations,
   static CA PEM, or only out-of-band external ownership.

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

Documentation-only change; no production code or generated manifests are
modified, so unit tests were not run.

Validation performed locally:

• All local Markdown links resolve.
• git diff --check passes.
• The signed-off commit is based directly on current upstream/master.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tariq-hasan]

Hi @shubhM13! Thanks for raising the PR. I have added some initial comments - mostly around the scope covered by the KEP.

[tariq-hasan]

I'm actually wondering if we'd want to deprecate webhook altogether - since the webhooks are now used not only for pod mutation but also for CRD mutation and validation.

[shubhM13]

Agree! the issue was closed as not planned #2502

[tariq-hasan]

It appears that a filesystem provider was already supported before v2 but was removed - #2016. The removal seems to be owed to the fact that bootstrap mechanism was prone to lead to failure modes - #855, #926, #1705, #1739, #1740, #2032, #2281.

Since the KEP addresses the various gaps by reshaping filesystem provision I am thinking this may warrant a callout in the motivation section.

[shubhM13]

Thanks! I’ll add a motivation callout distinguishing the previous mechanism from this proposal’s read-only external filesystem consumer and connect the relevant historical failures to the proposed safeguards.

[tariq-hasan]

At the same time I am wondering if we want to break down the scope of the KEP into multiple phases, and for phase 1, to only support filesystem serving identity with external CA ownership, and to retain full CA reconciliation as part of a phase 2 discussion. Would appreciate thoughts from @nabuskey @RobuRishabh @vjanelle @vikas-saxena02 as well.