feat(sparkconnect): add a Dependencies field to SparkConnectSpec

[Mr-Neutr0n]

Fixes #2959

Added a Dependencies struct and spec.dependencies field to SparkConnectSpec, and wired the Spark Connect controller to pass dependency flags through to start-connect-server.sh the same way SparkApplication does for spark-submit.

Local tests pass.

---

This change was prepared with AI assistance under human direction and review.

[github-actions]

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon!

Join the community:

• Slack: Join our #kubeflow-spark-on-kubernetes Slack channel.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mwielgus for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nabuskey]

You need to run make generate to auto generate the boilerplate code for added fields. Also we should match the field name to deps instead of dependencies.
Also, the passed in bash strings need to be handled like so: #3051

Let's add more tests too.

[Mr-Neutr0n]

Addressed the review: aligned Spark Connect with the SparkApplication deps field, shell-quoted all dependency arguments for the bash -c entrypoint, added shell-sensitive dependency coverage, and regenerated Go, CRD, OpenAPI, and Python API artifacts. d024dc8

[tariq-hasan]

/retitle feat(sparkconnect): add a Dependencies field to SparkConnectSpec

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

Hi @Mr-Neutr0n! I have added some more comments.

Please also update docs/website/user-guide/spark-connect.md, add a new examples/sparkconnect/spark-connect-dependencies.yaml and add/update e2e tests.

[Mr-Neutr0n]

Addressed the review in 244c6dd:

• Dependencies now carries only jars, packages, excludePackages, and repositories. files, pyFiles, and archives are gone from the API type, the controller, and the regenerated artifacts (deepcopy, OpenAPI, Swagger, CRDs, Python API). The type and field comments now describe the Spark Connect server instead of a Spark application.
• .spec.deps is documented in docs/website/user-guide/spark-connect.md.
• Added examples/sparkconnect/spark-connect-dependencies.yaml.
• Added test/e2e/sparkconnect_dependencies_test.go, which starts the server from the new example and asserts that the dependency flags reach spark-submit.

Verification: make unit-test (with envtest), make go-vet, make go-lint (0 issues) and a build of the e2e module all pass. I could not execute the e2e suite locally (no container runtime for kind here), so that test compiles but was not run in this environment.

[Mr-Neutr0n]

Two CI checks needed generated updates rather than code changes: the Python model was missing the generator's trailing blank lines, and docs/api-docs.md had to be regenerated for the new v1alpha1 type. I rebased onto current master, regenerated all artifacts on that base, and re-ran the checks locally (make unit-test, go vet, golangci-lint 0 issues, root and e2e builds). The e2e matrix had been cancelled by fail-fast on the previous run, so it should run fresh on this head.

[Mr-Neutr0n]

The e2e run was right: the dependencies test timed out waiting for the server pod to become ready, which needs the kind cluster to resolve the Maven artifacts (the rest of the suite only uses artifacts bundled in the Spark image). I narrowed the test to what this change owns: the SparkConnect is reconciled into a server pod whose spark-submit command carries the --jars, --packages, --exclude-packages, and --repositories flags. Happy to extend it to a full startup check if the e2e cluster has Maven egress.

[Mr-Neutr0n]

Rebased onto current master and resolved the conflicts on the same branch (agent/issue-2959-add-a-dependencies-fie, head ae912f8).

Conflicts were with the Spark Connect GPU work and #3188 (per-arg container args). dependenciesOption now emits --jars / --packages / --exclude-packages / --repositories as individual container arguments (no shell quoting), matching the new entrypoint; unit + e2e tests updated for that shape.

Review asks still hold: only jars/packages/excludePackages/repositories; sparkconnect-worded comments; docs/example/e2e coverage; regenerated deepcopy/OpenAPI/CRDs/API docs/Python models clean. go test ./internal/controller/sparkconnect/... passes. PR is MERGEABLE again — remaining BLOCKED is review/CI, not conflicts.