Add backend TOAST architecture documentation and clean up detoast access

- Add src/backend/access/common/README.toast documenting Plain TOAST and
  Direct TOAST formats, pointer structures (varatt_external vs. varatt_direct),
  catalog layouts, three-tier storage hierarchy (single chunk, flat multi-chunk,
  and hierarchical DAG), slicing traversal, and in-place upgrades.
- Add explanatory architectural comments to struct varatt_direct in varatt.h,
  DAG parameters in toast_internals.c, Direct TOAST catalog columns in toasting.c,
  and transparency notes in fmgr/README.
- Clean up redundant tuple descriptor natts check when fetching offsets_datum
  in toast_fetch_datum_direct_slice_recursive() in detoast.c, and expand the
  function documentation.
- Use an anonymous union for direct_tp and tp in ToastExternalMetadata to
  eliminate redundant memory and reflect the mutual exclusivity of external
  pointer formats.

Author: postsql

src/backend/access/common/README.toast

@@ -0,0 +1,115 @@
+src/backend/access/common/README.toast
+
+The Oversized-Attribute Storage Technique (TOAST)
+=================================================
+
+PostgreSQL stores table rows inside fixed-size pages (typically 8KB). When a
+row exceeds the target threshold (typically BLCKSZ / 4, or ~2KB), the storage
+engine uses TOAST to compress and/or move variable-length attributes (varlenas)
+out-of-line into a separate auxiliary relation, known as a TOAST table
+(cataloged as pg_toast_<relfilenumber>).
+
+PostgreSQL supports two on-disk external TOAST formats:
+1. Plain TOAST  (Traditional format, VARTAG_ONDISK = 18)
+2. Direct TOAST (Tree/TID direct format, VARTAG_DIRECT = 19)
+
+Whether a table uses Plain or Direct TOAST is controlled by the relation
+storage parameter 'toast_flavour' (or the GUC 'default_toast_flavour').
+
+
+1. Plain TOAST Format
+---------------------
+
+A Plain TOAST table has three columns:
+    (chunk_id OID, chunk_seq INT4, chunk_data BYTEA)
+
+Each out-of-line value is assigned a unique 32-bit OID (va_valueid). The
+value's data is broken into sequential chunks of up to TOAST_MAX_CHUNK_SIZE
+(typically ~2KB). An associated B-Tree index on (chunk_id, chunk_seq)
+indexes every chunk.
+
+The pointer stored in the main table's tuple is struct varatt_external (18 bytes):
+    int32   va_rawsize;     /* Original data size (includes 4-byte header) */
+    uint32  va_extinfo;     /* External saved size and 2 compression bits */
+    Oid     va_valueid;     /* Unique ID of value within TOAST table */
+    Oid     va_toastrelid;  /* RelID of TOAST table containing it */
+
+Detoasting requires opening an index scan on (chunk_id, chunk_seq). To read the
+entire datum or a partial slice, the executor searches the index for chunk 0,
+then iterates through consecutive chunks until the requested byte range is
+satisfied.
+
+
+2. Direct TOAST Format
+----------------------
+
+Direct TOAST eliminates index lookups during detoasting by embedding direct
+physical tuple identifiers (ItemPointerData / TID) into the pointer and chunk
+tuples.
+
+A Direct TOAST table has five columns:
+    (chunk_id OID, chunk_seq INT4, chunk_data BYTEA,
+     chunk_tids TID[], chunk_tid_offsets INT8[])
+
+The pointer stored in the main table's tuple is struct varatt_direct (18 bytes):
+    int32           va_rawsize;     /* Original data size (includes 4-byte header) */
+    uint32          va_extinfo;     /* External saved size and 2 compression bits */
+    Oid             va_toastrelid;  /* RelID of TOAST table containing it */
+    ItemPointerData va_tid;         /* Physical TID of root/terminal chunk */
+
+Notice that sizeof(varatt_direct) == sizeof(varatt_external) == 18 bytes.
+Because both pointer types have identical byte lengths, switching between
+Plain and Direct TOAST requires zero tuple header or layout adjustments in the
+parent table.
+
+2.1 Three-Tier Storage Hierarchy
+
+Depending on the size of the stored value, Direct TOAST structures chunks in one
+of three tiers:
+
+a) Single Chunk (datum size <= TOAST_MAX_CHUNK_SIZE, ~2KB):
+   The data fits into a single chunk tuple. The main tuple's va_tid points
+   directly to this chunk.
+   Detoasting: exactly 1 buffer page fetch, 0 index scans.
+
+b) Flat Multi-Chunk (<= DIRECT_TOAST_TREE_THRESHOLD = 100 chunks, up to ~200KB):
+   The data chunks (leaf chunks) are inserted first. A terminal "root" chunk is
+   then inserted which contains the last slice of data as chunk_data, plus a
+   chunk_tids array containing the ItemPointerData of all leaf chunks in sequence.
+   The main tuple's va_tid points to this root chunk.
+   Detoasting: fetches the root chunk, reads chunk_tids, and directly fetches each
+   referenced leaf chunk by TID without consulting any index.
+
+c) Hierarchical Tree / DAG (> DIRECT_TOAST_TREE_THRESHOLD = 100 chunks):
+   For very large values, chunks are organized into a multi-level balanced tree
+   with a fanout of DIRECT_TOAST_FANOUT (50).
+   Internal (non-leaf) chunks store:
+     - chunk_tids: array of ItemPointers to child chunks.
+     - chunk_tid_offsets: array of int64 cumulative start offsets for each child,
+       terminated by the total subtree size.
+   Detoasting: random-access partial slices (detoast_attr_slice) perform a binary
+   search over chunk_tid_offsets at each tree level, pruning subtrees that do not
+   overlap the requested range [sliceoffset, sliceoffset + slicelength). Slicing
+   runs in O(log N) buffer accesses without touching an index.
+
+2.2 Deletion & Memory Management
+
+When a tuple containing a Direct TOAST pointer is deleted or updated,
+toast_delete_datum_direct_recursive() traverses the chunk DAG by following
+chunk_tids recursively and deleting every referenced chunk tuple. Because all
+chunk locations are known from the embedded TIDs, no index lookups or table scans
+are required during cascaded deletion.
+
+2.3 Maintenance and Upgrades
+
+Direct TOAST tables still create and maintain the standard B-Tree index on
+(chunk_id, chunk_seq) to support VACUUM cleanup, sequential table scans, and
+fallback verification (such as amcheck).
+
+Existing tables can be upgraded in place from Plain to Direct TOAST:
+    ALTER TABLE my_table SET (toast_flavour = direct);
+or programmatically via pg_ensure_direct_toast(reloid).
+This operation adds chunk_tids and chunk_tid_offsets to the TOAST table with
+fast-default NULLs without rewriting table rows or requiring exclusive locks on
+large data sets. Existing Plain TOAST pointers continue to be read via the
+traditional path, while new out-of-line writes generate Direct TOAST pointers.

src/backend/access/common/detoast.c

@@ -665,7 +665,20 @@ toast_slice_copy_chunk(struct varlena *result, const char *chunk_data,
 }
 
 /*
- * Recursively traverse and fetch slices from a direct TOAST tree/DAG.
+ * toast_fetch_datum_direct_slice_recursive -
+ *
+ * Traverse a Direct TOAST tree/DAG to retrieve full datums or partial slices.
+ *
+ * Direct TOAST organizes chunks either as a single chunk, a flat multi-chunk
+ * list (chunk_tids populated, chunk_tid_offsets NULL), or a multi-level tree
+ * (both chunk_tids and chunk_tid_offsets populated).
+ *
+ * - Leaf chunks (chunk_tids IS NULL) copy their slice payload directly into
+ *   the result buffer at *logical_offset.
+ * - Flat multi-chunk roots iterate through all child TIDs in chunk_tids.
+ * - Interior tree chunks inspect chunk_tid_offsets to prune any subtrees that
+ *   do not overlap the requested range [sliceoffset, sliceoffset + slicelength),
+ *   achieving O(log N) slice fetching without consulting an index.
  */
 static void
 toast_fetch_datum_direct_slice_recursive(Relation toastrel, ItemPointer tid,
@@ -711,8 +724,7 @@ toast_fetch_datum_direct_slice_recursive(Relation toastrel, ItemPointer tid,
 		int			nelems;
 		int			i;
 
-		if (slot->tts_tupleDescriptor->natts >= 5)
-			offsets_datum = slot_getattr(slot, 5, &is_null_offsets);
+		offsets_datum = slot_getattr(slot, 5, &is_null_offsets);
 
 		deconstruct_array_builtin(arr, TIDOID, &elems, &nulls, &nelems);
 

src/backend/access/common/toast_internals.c

@@ -30,6 +30,20 @@
 
 int			toast_flavour = TOAST_FLAVOUR_PLAIN;
 
+/*
+ * Direct TOAST chunk organization parameters:
+ *
+ * - DIRECT_TOAST_TREE_THRESHOLD (100):
+ *   Values requiring up to 100 chunks (~200KB) are structured with a flat
+ *   root chunk containing a single chunk_tids array of all leaf chunk TIDs.
+ *   This avoids tree depth overhead for small to medium values.
+ *
+ * - DIRECT_TOAST_FANOUT (50):
+ *   When a value exceeds DIRECT_TOAST_TREE_THRESHOLD, a multi-level balanced
+ *   tree is constructed where intermediate index chunks hold up to 50 child
+ *   TIDs (chunk_tids) and cumulative byte boundaries (chunk_tid_offsets).
+ *   This enables O(log N) slice retrieval without index scans.
+ */
 #define DIRECT_TOAST_TREE_THRESHOLD	100
 #define DIRECT_TOAST_FANOUT			50
 

src/backend/catalog/toasting.c

@@ -272,6 +272,12 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
 					   "chunk_data",
 					   BYTEAOID,
 					   -1, 0);
+	/*
+	 * Direct TOAST columns:
+	 * chunk_tids stores child chunk TIDs for flat multi-chunk roots and interior DAG nodes.
+	 * chunk_tid_offsets stores byte offsets within chunk_tids for binary-search slicing.
+	 * Both columns are NULL for simple leaf chunks or legacy plain TOAST rows.
+	 */
 	TupleDescInitEntry(tupdesc, (AttrNumber) 4,
 					   "chunk_tids",
 					   TIDARRAYOID,
@@ -361,6 +367,10 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
 	 * duplicate TOAST chunk OIDs. The index might also be a little more
 	 * efficient this way, since btree isn't all that happy with large numbers
 	 * of equal keys.
+	 *
+	 * Even when Direct TOAST is active (which fetches chunks directly by TID),
+	 * this index is still created and maintained for backward compatibility,
+	 * sequential scan fallback, and VACUUM validation.
 	 */
 
 	indexInfo = makeNode(IndexInfo);

src/backend/utils/fmgr/README

@@ -206,7 +206,9 @@ For TOAST-able data types, the PG_GETARG macro will deliver a de-TOASTed
 data value.  There might be a few cases where the still-toasted value is
 wanted, but the vast majority of cases want the de-toasted result, so
 that will be the default.  To get the argument value without causing
-de-toasting, use PG_GETARG_RAW_VARLENA_P(n).
+de-toasting, use PG_GETARG_RAW_VARLENA_P(n).  Whether the out-of-line
+value is stored using Plain TOAST or Direct TOAST is an internal storage
+detail handled transparently by detoast_attr().
 
 Some functions require a modifiable copy of their input values.  In these
 cases, it's silly to do an extra copy step if we copied the data anyway

src/include/varatt.h

@@ -82,6 +82,19 @@ VARATT_EXTERNAL_OID8_SET_VALUEID(varatt_external_oid8 *toast_pointer, Oid8 id)
 	toast_pointer->va_valueid_hi = (uint32) (id >> 32);
 }
 
+/*
+ * varatt_direct is a "Direct TOAST pointer".
+ *
+ * Instead of identifying chunks via an OID (va_valueid) which requires a
+ * B-Tree index scan on (chunk_id, chunk_seq), va_tid points directly to the
+ * root/terminal chunk tuple on disk.
+ *
+ * Notice that sizeof(varatt_direct) == sizeof(varatt_external) == 18 bytes.
+ * This ensures that switching to Direct TOAST causes no size difference or
+ * alignment change in parent table tuples.
+ *
+ * Like varatt_external, this struct is stored unaligned within actual tuples.
+ */
 typedef struct varatt_direct
 {
 	int32		va_rawsize;		/* Original data size (includes header) */