fix(databricks): use value check for databricks_connect_use_serverless

[gabepesco]

Previously, _set_spark_engine_adapter_if_needed used a key-presence check ("databricks_connect_use_serverless" in self._extra_config) to decide whether to route to serverless. This caused cluster_id to be ignored whenever the key existed in config, even if its value was False.

Changed to a value check (self._extra_config.get(...)) so that cluster_id is correctly used when serverless is explicitly disabled.

Closes #5842

Description

When a user sets databricks_connect_use_serverless: false in their config alongside a databricks_connect_cluster_id, the adapter was ignoring the cluster ID and routing all Databricks Connect sessions to serverless anyway. This is because the routing check in _set_spark_engine_adapter_if_needed tested for the presence of the key in _extra_config rather than its value.

The fix changes the check from:

if "databricks_connect_use_serverless" in self._extra_config:

to:

if self._extra_config.get("databricks_connect_use_serverless"):

This is consistent with how the same config key is read elsewhere in the codebase (e.g. _use_spark_session and connection.py).

Test Plan

Three new unit tests added to tests/core/engine_adapter/test_databricks.py:

• test_databricks_connect_routes_to_cluster_id — verifies cluster_id is passed to DatabricksSession.builder.remote when the serverless flag is absent
• test_databricks_connect_routes_to_serverless — verifies serverless=True is passed when the flag is True
• test_databricks_connect_cluster_id_not_overridden_by_falsy_serverless — the regression case: verifies cluster_id is still used when the flag is explicitly False

Checklist

• I have run make style and fixed any issues
• I have added tests for my changes (if applicable)
• All existing tests pass (make fast-test)
• My commits are signed off (git commit -s) per the DCO