feat: add operator flags to inject default pod labels and annotations

[Fumer057]

Fixes #3148

Description

This PR adds support for platform administrators to inject operator-level default labels and annotations onto every Spark driver and executor pod.

Changes:

• Adds two new CLI flags to the controller: --default-pod-labels and --default-pod-annotations (both accept JSON strings)
• Adds corresponding Helm chart values under .Values.spark.defaultPodLabels\ and .Values.spark.defaultPodAnnotations\
• Merges the global defaults with per-app labels/annotations before submitting Spark applications, ensuring that user-provided labels on the \SparkApplication\ spec correctly override the global defaults.
• Applies the global defaults directly to Spark Connect server pods upon creation.

Testing Done

• Added comprehensive unit tests in \pkg/util/sparkapplication_test.go\ verifying default label application and user override behaviors.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mwielgus for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon!

Join the community:

• Slack: Join our #kubeflow-spark-on-kubernetes Slack channel.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

Hi @Fumer057! Thanks for raising the PR. I have added suggestions for changes.

Also please note that operator-reserved labels/annotations should not be overwritten by admin-controlled operator flags. Please add this constraint to the implementation as well.

And also please add user documentation for the feature in docs/website/user-guide/writing-sparkapplication.md and docs/website/user-guide/spark-connect.md.

[Fumer057]

Hi @tariq-hasan, thanks for the detailed review! I've addressed all of your feedback in the latest commit:

• Removed | quote from the Helm deployment template.
• Added K8s API validation in start.go for the label/annotation keys and values, and added logic to block the operator-reserved prefix (sparkoperator.k8s.io/).
• Wired up the SparkConnect executor pods to receive the default labels and annotations.
• Added Helm unit tests for the new flags.
• Updated the writing-sparkapplication.md and spark-connect.md user guides.
• Regenerated the Helm chart README.md via make helm-docs.

Let me know if there's anything else needed!

[Fumer057]

@tariq-hasan I have implemented this. SparkConnect executor pods are now properly wired up to receive the default labels and annotations.

[tariq-hasan]

Hi @Fumer057! Thanks for the contributions. I have added a few more comments.

[Fumer057]

@tariq-hasan I have addressed all the new review comments in the latest commit:

• Moved the variable declarations and flags for \default-pod-labels\ and \default-pod-annotations\ to be right below \default-service-account.
• Replaced the manual label/annotation validation with \k8s.io/apimachinery/pkg/apis/meta/v1/validation\ and \k8s.io/apimachinery/pkg/api/validation.
• Blocked Spark-reserved labels from being overridden via operator defaults and added missing label constants to \pkg/common/spark.go.
• Removed driver and executor --conf\ injection from SparkConnect client mode and directly injected labels/annotations onto the executor pod template instead.
• Added \defaultPodLabels\ and \defaultPodAnnotations\ to the \logger.Info\ call in the SparkApplication controller.
• Updated \�alues.yaml\ comments to use 'CR' instead of 'SparkApplication' and ran \make helm-docs.
• Fixed the precedence rule in \pkg/util/sparkapplication.go\ to apply defaults onto the pod \Template\ directly instead of at the spec-level labels/annotations, and explicitly skipped any keys already defined at higher levels of precedence.

Let me know if any further tweaks are needed!