SVNKit MapDB cache file corrupted after ungraceful JVM shutdown — file handle leaked causing WAITING thread and preventing cache deletion

[Krivoy05]

Jenkins and plugins versions report

Environment

Jenkins: 2.555.1
OS: Windows Server 2019 - 10.0
Java: 21.0.3 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
Office-365-Connector:5.3.0
analysis-model-api:14.13.0-998.v6dec20409ce6
ansible:635.v34b_48e979c86
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6.1-195.v65ffe15189a_d
artifactory:4.0.8
artifactory-artifact-manager:320.v6d1b_a_84b_16db_
artifactory-client-api:2.21.2-166.v56e395cd41e0
asm-api:9.10.1-216.va_9256d3b_844b_
authentication-tokens:1.144.v5ff4a_5ec5c33
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-1048.va_c299057e35c
bouncycastle-api:2.30.1.84-291.v9f17b_21896e2
branch-api:2.1280.v0d4e5b_b_460ef
build-monitor-plugin:1076.v651a_12e63830
build-timeout:1.40
build-user-vars-plugin:214.va_eed2ed849ca_
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:415.vf022234a_931d
cloud-stats:423.v63f017a_f582c
cloudbees-bitbucket-branch-source:937.3.2
cloudbees-folder:6.1100.ve9eed61d16c4
clover:4.15.0.677.v22c71ca_e6450
command-launcher:134.v025a_5fcf9dea_
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-82.v4df6c060eb_66
commons-httpclient3-api:3.1-3
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
concurrent-step:1.0.0
config-file-provider:1013.v73c323e52b_1f
copyartifact:795.ve8e151429b_27
credentials:1502.v5c95e620ddfe
credentials-binding:725.ve52b_2328a_fde
dark-theme:652.vea_da_dfea_e769
data-tables-api:2.3.8-1570.v1cb_1cd2a_0fb_c
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:472.vee120e23d3a_c
docker-java-api:3.7.1-136.v5d70f77a_c3d6
docker-plugin:1316.v75635a_002b_0a_
docker-slaves:1.0.7
docker-workflow:634.vedc7242b_eda_7
downstream-build-cache:1.7
dtkit-api:3.0.3
durable-task:671.v340ff7959010
echarts-api:6.0.0-1287.vfd24c22a_3d00
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:2038.v7b_8817a_499d9
extensible-choice-parameter:255.vfa_41f46828ef
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.263.v941d21defef7
font-awesome-api:7.2.0-990.vf220b_2a_496f9
forensics-api:4.1891.v5e60f3377506
generic-webhook-trigger:2.4.2
git:5.10.1
git-client:6.6.0
git-server:137.ve0060b_432302
github:1.47.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1967.1969.v205fd594c821
gradle:2.19.1244.v1f9866817fec
groovy:537.v741a_5a_f1b_581
gson-api:2.14.0-201.v8eefe5515533
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427.1
http_request:1.25
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson-annotations2-api:2.22-19.v10a_a_582ea_26e
jackson2-api:2.21.2-436.v29efdb_7418ff
jackson3-api:3.2.0-83.vfb_04cee1ea_0f
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.9-19.v2b_a_5b_44d9a_1c
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jersey2-api:2.48-180.ve47b_264f849b_
jfrog:1.8.0
jjwt-api:0.13.0-141.vd58b_a_9592b_6c
joda-time-api:2.14.2-193.v422b_efce56e0
jquery3-api:3.7.1-687.v68d468e40b_30
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20260522-217.v0b_18b_8cd4672
json-path-api:3.0.0-218.vcd4dd1355de2
jsoup:1.22.2-95.vc5d00f1eb_42d
junit:1413.v736fa_5b_61d80
ldap:807.809.vd3a_4e5e4ec98
locale:641.v84f22d75fd8b_
lockable-resources:1524.v2c727b_b_e56ef
mail-watcher-plugin:1.21
mailer:534.v1b_36f5864073
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.10
matrix-combinations-parameter:1.3.3
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.17.1-187.v0341274c2905
mina-sshd-api-core:2.17.1-187.v0341274c2905
monitoring:2.8.0
msbuild:1.42
nodelabelparameter:851.vd94e5048d321
okhttp-api:5.3.2-200.vedb_720a_cf1f8
oss-symbols-api:463.vff8a_5631a_09d
pam-auth:1.12
parameter-separator:334.v1fc0e534c71d
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:943.v4c6b_7fda_f02d
pipeline-groovy-lib:798.v5cc688825312
pipeline-input-step:551.vdff487c5998c
pipeline-milestone-step:152.v6e22b_8cfc66c
pipeline-model-api:2.2291.v2934911987b_6
pipeline-model-definition:2.2291.v2934911987b_6
pipeline-model-extensions:2.2291.v2934911987b_6
pipeline-rest-api:2.41
pipeline-stage-step:345.va_96187909426
pipeline-stage-tags-metadata:2.2291.v2934911987b_6
pipeline-stage-view:2.41
pipeline-utility-steps:3.810.va_7672d206740
plain-credentials:199.v9f8e1f741799
plot:2.4.0
plugin-util-api:7.1341.v039f146993d9
prism-api:1.30.0-741.v034eb_0b_0a_a_fa_
publish-over:238.v1db_583004c9b_
publish-over-cifs:200.v1b_fb_139a_b_5a_8
pubsub-light:1.19
resource-disposer:0.25
scm-api:728.vc30dcf7a_0df5
script-security:1402.v94c9ce464861
sidebar-link:2.5.0
simple-theme-plugin:230.v8b_fd91b_b_800c
snakeyaml-api:2.5-149.v72471e9c6371
snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e
sonar:2.18.3
sse-gateway:1.29
ssh-credentials:372.va_250881b_08cd
ssh-slaves:3.1097.v868116049892
sshd:3.384.vc89b_5e138cf9
startup-trigger-plugin:2.9.4
structs:362.va_b_695ef4fdf9
subversion:1303.vcfd9679fb_c12
swarm:1254.vf26a_5e188f26
test-results-analyzer:309.vda_3a_a_f100542
theme-manager:346.v06cca_64c6a_37
thinBackup:2.1.4
thycotic-secret-server:2.1.1
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
uno-choice:2.8.9
variant:70.va_d9f17f859e0
warnings-ng:13.10097.v277a_958a_b_c09
woodstox-core-api:7.2.1-6.v3718a_a_11f5c4
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1413.v2ff1a_5e720fa_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4331.v9d06ed4658ff
workflow-durable-task-step:1479.v56e587f413a_7
workflow-job:1571.1580.v18e46842c125
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:724.v538c2362b_dfb_
workflow-support:1015.v785e5a_b_b_8b_22
ws-cleanup:0.49
xunit:3.1.6
yet-another-build-visualizer:1.17

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Windows Server 2019/2022 (x64)
Agents: Windows 10 / Windows 11 (x64), connected via JNLP4

Reproduction steps

1. Jenkins is running on Windows with the Subversion plugin installed.

2. A Multibranch Pipeline is configured with an SVN repository as the SCM source.
   Branch Indexing runs periodically via SCMTrigger.

3. SVNKit opens a MapDB cache file located at:
   $JENKINS_HOME/caches/svn-.db
   (along with .db.p and .db.t companion files)

4. Jenkins JVM is terminated ungracefully — in our case caused by
   ForeScout SecureConnector (a corporate NAC agent) sending SIGKILL
   instead of allowing graceful SIGTERM shutdown.
   Log evidence:
   "SIGINT to failed - Killing as fallback"

5. Jenkins restarts. SVNKit attempts to reopen the MapDB cache file.
   MapDB detects a bad checksum and throws:
   java.io.IOException: Wrong index checksum,
   store was not closed properly and could be corrupted.

6. The exception propagates up but the file handle on .db is NOT released.
   The SVNKit thread enters WAITING state indefinitely:
   Thread: pool-1-svnkit-thread-1 | State: WAITING

7. Branch Indexing fails with FATAL error:
   java.io.IOException: Wrong index checksum,
   store was not closed properly and could be corrupted.
   at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:...)
   at org.mapdb.StoreDirect.(StoreDirect.java:...)

8. The corrupted .db files cannot be deleted while Jenkins is running
   because pool-1-svnkit-thread-1 holds the file handle (Windows file lock).

9. Workaround: calling System.gc() + System.runFinalization() from
   Script Console forces the GC to finalize the SVNKit object,
   which releases the file handle, after which deletion succeeds.

Expected Results

When MapDB detects a corrupted cache file on startup, SVNKit should:

1. Catch the IOException from MapDB properly, ensuring the file handle
   is closed (try-with-resources or explicit close() in finally block).

2. Automatically delete the corrupted cache files and recreate them
   from scratch, allowing Branch Indexing to continue without manual
   intervention.

3. The SVNKit thread should NOT enter an indefinite WAITING state
   due to an unreleased lock on a corrupted file.

Actual Results

When MapDB throws IOException due to bad checksum after ungraceful shutdown:

1. The file handle on svn-.db is NOT released by SVNKit.
   Thread pool-1-svnkit-thread-1 enters indefinite WAITING state.

2. Branch Indexing fails permanently with FATAL IOException on every run
   until the cache files are manually deleted.

3. On Windows, the file lock prevents any external deletion attempt
   (File.delete() returns false) while Jenkins is running.

4. The only workaround is:
   a) Force JVM GC via Script Console to release the handle, then delete, OR
   b) Stop the Jenkins service, delete files, restart Jenkins.

Full stack trace from Jenkins log:
FATAL o.j.p.b.multibranch.BranchIndexingCause
java.io.IOException: Wrong index checksum,
store was not closed properly and could be corrupted.
at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:207)
at org.mapdb.StoreDirect.(StoreDirect.java:189)
at org.mapdb.StoreWAL.(StoreWAL.java:74)
at org.mapdb.DBMaker.make(DBMaker.java:701)
at jenkins.scm.impl.subversion.SVNRepositoryView.(...)

Anything else?

Root cause chain:
ForeScout SecureConnector (corporate NAC agent)
-> sends SIGKILL to JVM (no graceful shutdown)
-> MapDB WAL not flushed, checksum not written
-> SVNKit catches IOException but leaks file handle
-> Windows file lock prevents deletion
-> Branch Indexing fails on every subsequent run

The core bug is in SVNKit / Subversion Plugin:
IOException from MapDB is not handled with proper resource cleanup.
The file handle must be closed in a finally{} block or
try-with-resources regardless of whether the DB opened successfully.

Suggested fix location:
SVNRepositoryView constructor or wherever DBMaker.make() is called —
wrap in try-catch and ensure db.close() or file.delete() is called
on IOException before rethrowing.

Environment note:
This is reproducible on any Windows Jenkins instance where the JVM
can be killed ungracefully (Windows Update reboot, antivirus/NAC
agents, OOM killer, power loss, etc.).
Linux is less affected due to different file locking semantics,
but the leaked handle and corrupted cache issue remains.

Workaround applied:
Scheduled nightly cleanup job running System.gc() + delete()
via Script Console to prevent accumulation of corrupted cache files.

Are you interested in contributing a fix?

No

[Krivoy05]

I didn't get your answer. I did not add any file

[Eccenux]

That seems like an SVNKit issue, and this is not an SVNKit issue tracker. See: https://svnkit.com/

Also, after an ungraceful JVM shutdown as described above, I would probably just remove the repo anyway or fix it outside of Jenkins.