Skip to content

Rewrite large object migration as a direct relation copy. - #56

Open
ibrarahmad wants to merge 1 commit into
lo-harden-lifecyclefrom
lo-migrate-storage-rewrite
Open

ibrarahmad wants to merge 1 commit into
lo-harden-lifecyclefrom
lo-migrate-storage-rewrite

Conversation

@ibrarahmad

@ibrarahmad ibrarahmad commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Based on #55.

Migration used to rewrite every large object through the LO API, which drops things the API cannot carry. It is now a direct copy of tuples between the native and lolor relations, in C (lolor.migrate_storage(), src/lolor_migrate.c). The layouts match by construction and are checked at run time.

What that fixes: comments were lost (now parked in lolor.pg_largeobject_description and put back); ownership was set with a raw UPDATE, so DROP ROLE, REASSIGN OWNED and DROP OWNED did not see migrated objects (now recorded in pg_shdepend); sparse objects were zero-filled (pages are copied as is); native objects are removed in batches through performMultipleDeletions().

Migration holds ShareRowExclusiveLock on both stores until commit, so a concurrent lo_write() cannot land between the copy and the delete and be lost. Locks are taken in a fixed order.

Both migrations also refuse while any other session is connected to the database, for the same reason enable() and disable() do in #55. The concurrent-write test in t/010 is now a refusal test, and the lock is checked from the migrating session itself.

Also: migrate_to_native() no longer needs lolor enabled, so DROP EXTENSION works in the disabled state; migration refuses on security labels and on objects that refer to a dropped role, with a clear message instead of "role N was concurrently dropped"; OID conflicts list the OIDs; lolor.digest() and lolor.native_lo_oids() with a peer_oids check for cross-node collisions; lo_unlink() removes a parked comment so a reused OID cannot inherit it. The README no longer suggests running the migration through spock.replicate_ddl(), since that runs it inside apply workers.

1.3.0 is unreleased, so the SQL goes into lolor--1.2.2--1.3.0.sql.

Tests: t/009_migration.pl and t/010_migration_locking.pl, plus regression coverage for sparse objects, comments, pg_shdepend, orphans and the helpers.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 85b7ec76-6ae4-4f4a-b660-20a0b39318c8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@codacy-production

codacy-production Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 16 high · 5 medium

Results:
21 new issues

Category Results
Compatibility 16 high (4 false positives)
Complexity 5 medium

View in Codacy

🟢 Metrics 67 complexity · 2 duplication

Metric Results
Complexity 67
Duplication 2

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from 6088141 to ad9862d Compare September 16, 2026 14:01
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from ad9862d to c3d60b9 Compare September 22, 2026 06:17
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from c3d60b9 to 1f7b1cf Compare September 22, 2026 06:29
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from 1f7b1cf to b5f7d0f Compare September 22, 2026 06:36
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from b5f7d0f to b553171 Compare September 22, 2026 06:40
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from b553171 to da6520b Compare September 22, 2026 09:55
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from da6520b to a0a9408 Compare September 23, 2026 16:03
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from a0a9408 to 9b1bf14 Compare September 23, 2026 19:50
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch 2 times, most recently from 54f085b to 04a66f0 Compare September 24, 2026 05:06
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from 04a66f0 to b49849f Compare September 24, 2026 06:27
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from b49849f to 02a1772 Compare September 28, 2026 15:52
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from 02a1772 to 48d51e1 Compare September 29, 2026 09:59
Migration rewrote every object through the large object API, which lost
information the API cannot carry.  Copy tuples between the two relations
instead: their layouts match by construction, and that layout is now
verified at run time rather than assumed.

Comments were silently discarded, having nowhere to live while an object
sits in lolor storage; they are parked in lolor.pg_largeobject_description
and reinstated on the way back.  Ownership was applied with a raw catalog
UPDATE, leaving no pg_shdepend row, so DROP ROLE succeeded on a role that
still owned migrated objects and DROP OWNED could delete another user's;
the dependencies are now recorded properly.  Pages are copied verbatim, so
a sparse 10 MB object no longer materialises as 10 MB of pages.  Native
objects go through performMultipleDeletions(), the path DROP uses, so core
removes their dependencies, comments and labels.

Hold ShareRowExclusiveLock on both stores until commit.  Large object
access takes RowExclusiveLock, which does not conflict with itself, so a
concurrent lo_write() could commit between the page copy and the emptying
of the source store and be lost without trace.  The locks are taken in the
same order in either direction, so opposing migrations cannot deadlock.

migrate_to_native() no longer calls the renamed _orig functions, so it
works whether or not lolor is enabled, and DROP EXTENSION in the disabled
state now rescues the objects instead of failing.  Security labels cannot
be represented in lolor storage and cannot be replayed without their
provider, so migration refuses rather than discarding them.

Add lolor.digest() and lolor.native_lo_oids() with a peer_oids pre-flight,
which turns the documented cross-node OID collision hazard into a refusal.
@ibrarahmad
ibrarahmad force-pushed the lo-migrate-storage-rewrite branch from 48d51e1 to 2ea271d Compare September 29, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant