Skip to content

[spark] Add paimon spark4.2 module #9265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LuciferYang wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[spark] Add paimon spark4.2 module #9265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f37fb20
[spark] Add paimon spark4.2 module
LuciferYang Aug 13, 2026
a2bad68
[spark] Fix three test failures on the Spark 4.2 baseline
LuciferYang Aug 17, 2026
a7698f0
[spark] Strip the explicit UTF8_BINARY collation from SHOW CREATE TAB…
LuciferYang Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/publish_snapshot-jdk17.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ jobs:
echo "<password>$ASF_PASSWORD</password>" >> $tmp_settings
echo "</server></servers></settings>" >> $tmp_settings

mvn --settings $tmp_settings -ntp clean install -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 -am
mvn --settings $tmp_settings -ntp clean install -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 -am
# skip deploy paimon-spark-common_2.13 since they are already deployed in publish-snapshot.yml
mvn --settings $tmp_settings -ntp clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-ut_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13
mvn --settings $tmp_settings -ntp clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-ut_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13

rm $tmp_settings
8 changes: 4 additions & 4 deletions .github/workflows/release-java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ jobs:
mvn -ntp -B "${enforcer_goal}" \
-Denforcer.rules="${enforcer_rules}" \
-Papache-release,docs-and-source,spark4 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 \
-am
;;
*)
Expand Down Expand Up @@ -186,22 +186,22 @@ jobs:
jdk17)
capture_expected_projects \
-Papache-release,docs-and-source,spark4 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13
# This install only supplies reactor dependencies for the deploy
# below. Their Javadocs are built by the owning release lane.
mvn clean install -ntp -B \
-Pdocs-and-source,spark4 \
-DskipTests -Dmaven.javadoc.skip=true \
-Dstyle.color=never \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1,paimon-spark/paimon-spark-4.2 \
-am \
2>&1 | tee "${log}"
mvn deploy -ntp -B \
-Papache-release,docs-and-source,spark4 \
-DskipTests -Dgpg.skip=true -Dstyle.color=never \
-DdeployAtEnd=true \
-DaltDeploymentRepository="${alt_repository}" \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 \
2>&1 | tee -a "${log}"
;;
*)
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/utitcase-spark-4.x.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
jvm_timezone=$(random_timezone)
echo "JVM timezone is set to $jvm_timezone"
test_modules=""
for suffix in ut 4.0 4.1; do
for suffix in ut 4.0 4.1 4.2; do
test_modules+="org.apache.paimon:paimon-spark-${suffix}_2.13,"
done
test_modules="${test_modules%,}"
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/ecosystem/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ under the License.
| Engine | Version | Batch Read | Batch Write | Create Table | Alter Table | Streaming Write | Streaming Read | Batch Overwrite | DELETE & UPDATE | MERGE INTO | Time Travel |
|:-------------------------------------------------------------------------------:|:-------------:|:-----------:|:-----------:|:-------------:|:-------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------:|:-----------:|
| Flink | 1.16 - 1.20 | ✅ | ✅ | ✅ | ✅(1.17+) | ✅ | ✅ | ✅ | ✅(1.17+) | ❌ | ✅ |
| Spark | 3.2 - 4.1 | ✅ | ✅ | ✅ | ✅ | ✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅ | ✅(3.3+) |
| Spark | 3.2 - 4.2 | ✅ | ✅ | ✅ | ✅ | ✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅ | ✅(3.3+) |
| Hive | 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Trino | 420 - 440 | ✅ | ✅(427+) | ✅(427+) | ✅(427+) | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Presto | 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/project/download.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ This documentation is a guide for downloading Paimon Jars.
| Flink 1.17 | [paimon-flink-1.17-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/) |
| Flink 1.16 | [paimon-flink-1.16-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/) |
| Flink Action | [paimon-flink-action-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/@@VERSION@@/) |
| Spark 4.2 | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/) |
| Spark 4.1 | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/) |
| Spark 4.0 | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/) |
Expand Down Expand Up @@ -76,6 +77,7 @@ This documentation is a guide for downloading Paimon Jars.
| Flink 1.17 | [paimon-flink-1.17-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/paimon-flink-1.17-@@VERSION@@.jar) |
| Flink 1.16 | [paimon-flink-1.16-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/paimon-flink-1.16-@@VERSION@@.jar) |
| Flink Action | [paimon-flink-action-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/@@VERSION@@/paimon-flink-action-@@VERSION@@.jar) |
| Spark 4.2 | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/paimon-spark-4.2_2.13-@@VERSION@@.jar) |
| Spark 4.1 | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/paimon-spark-4.1_2.13-@@VERSION@@.jar) |
| Spark 4.0 | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/paimon-spark-4.0_2.13-@@VERSION@@.jar) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/paimon-spark-3.5_2.12-@@VERSION@@.jar) |
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/project/verifying-a-release-candidate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ the packaging-equivalent build without reproducing the full JDK 17 test lane:
(
cd "paimon-${PAIMON_VERSION}"
mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1,paimon-spark/paimon-spark-4.2 \
-am
)
```
Expand Down
7 changes: 6 additions & 1 deletion docs/docs/spark/quick-start.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ under the License.

Paimon supports the following Spark versions with their respective Java and Scala compatibility. We recommend using the latest Spark version for a better experience.

- Spark 4.x (including 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13
- Spark 4.x (including 4.2, 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13

- Spark 3.x (including 3.5, 3.4, 3.3, 3.2) : Pre-built with Java 8 and Scala 2.12/2.13

Expand All @@ -45,6 +45,7 @@ Download the jar file with corresponding version.

| Version | Jar (Scala 2.12) | Jar (Scala 2.13) |
|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.2 | - | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/paimon-spark-4.2_2.13-@@VERSION@@.jar) |
| Spark 4.1 | - | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/paimon-spark-4.1_2.13-@@VERSION@@.jar) |
| Spark 4.0 | - | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/paimon-spark-4.0_2.13-@@VERSION@@.jar) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/paimon-spark-3.5_2.12-@@VERSION@@.jar) | [paimon-spark-3.5_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/paimon-spark-3.5_2.13-@@VERSION@@.jar) |
Expand All @@ -62,6 +63,7 @@ Download the jar file with corresponding version.

| Version | Jar (Scala 2.12) | Jar (Scala 2.13) |
|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.2 | - | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/) |
| Spark 4.1 | - | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/) |
| Spark 4.0 | - | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/) | [paimon-spark-3.5_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/) |
Expand Down Expand Up @@ -89,6 +91,9 @@ mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.0 -am -Pspark4

# build paimon spark 4.1
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.1 -am -Pspark4

# build paimon spark 4.2
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.2 -am -Pspark4
```

For Spark 3.5, you can find the bundled jar in `./paimon-spark/paimon-spark-3.5/target/paimon-spark-3.5_2.12-@@VERSION@@.jar`.
Expand Down
6 changes: 5 additions & 1 deletion paimon-spark/paimon-spark-4.0/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,13 @@ under the License.

<!-- test -->

<!-- The 4.0-compiled build of `paimon-spark-ut`'s test sources. The baseline build cannot
be used here: Spark 4.2 turned `QueryTest` into a trait (SPARK-55910), so a
4.2-compiled `PaimonSparkTestBase` carries it as an interface and fails to load on a
4.0/4.1 runtime. See `paimon-spark-ut-4.0/pom.xml`. -->
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-spark-ut_${scala.binary.version}</artifactId>
<artifactId>paimon-spark-ut-4.0_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
Expand Down
Loading
Loading