From 8ae2fe3fe3f17d2e2b207cd392ec5f6a05bba73a Mon Sep 17 00:00:00 2001 From: bram-atmire Date: Sat, 12 Sep 2026 10:10:52 +0200 Subject: [PATCH] Document staged uploads and script invocation from staged uploads Describe the generic staging resource under /api/core/uploads: metadata preflight against the multipart file-size limit, bounded chunk requests, committed-progress reads, cancellation, states and the recorded result that makes a repeated consuming request idempotent. Document the JSON form of POST /api/system/scripts/{name}/processes that attaches staged uploads as input files, and list the resource in the endpoint index. Relates to DSpace/DSpace#13103 and DSpace/DSpace#13106. Claude-Session: https://claude.ai/code/session_01HtPYmqfFzg7fmZgovGWP5A --- endpoints.md | 1 + scripts-endpoint.md | 35 +++++++++ uploads.md | 172 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 208 insertions(+) create mode 100644 uploads.md diff --git a/endpoints.md b/endpoints.md index ba57c17e..8b17348c 100644 --- a/endpoints.md +++ b/endpoints.md @@ -10,6 +10,7 @@ * [/api/core/bitstreams](bitstreams.md) * [/api/core/bitstreamformats](bitstreamformats.md) * [/api/core/bundles](bundles.md) +* [/api/core/uploads](uploads.md) * [/api/core/metadatafields](metadatafields.md) * [/api/core/metadataschemas](metadataschemas.md) * [/api/core/edititems](edititems.md) diff --git a/scripts-endpoint.md b/scripts-endpoint.md index a1c264fe..49d81070 100644 --- a/scripts-endpoint.md +++ b/scripts-endpoint.md @@ -182,6 +182,7 @@ Status codes: * 404 Resource Not Found - if a script with the specified name doesn't exist ## Script Invocation + **POST /api/system/scripts/<:script-name>/processes** POST requests to this endpoint will start the corresponding script with the provided parameters. All parameter values should be provided in the body that has to use the `multipart/form-data` content type. Once the upload is complete and the script was started successfully, this endpoint will return details on the scripts execution @@ -226,3 +227,37 @@ Status codes: * 400 Not found - if the provided parameters don't match the script expectations * 404 Not found - if the script doesn't exist * 413 Payload too large - uploaded file is larger than limit set in configuration parameter `spring.servlet.multipart.max-file-size` + +### Script Invocation from staged uploads + +**POST /api/system/scripts/<:script-name>/processes** + +The same endpoint also accepts an `application/json` body, which lets a client attach input files +that were previously [staged in bounded chunks](uploads.md) instead of sending one multipart request: + +```json +{ + "properties": [ + { "name": "--add" }, + { "name": "--zip", "value": "batch.zip" }, + { "name": "-v" }, + { "name": "--collection", "value": "954e5cfa-6990-4c85-ae42-f30d8c7888e2" } + ], + "uploads": [ "18c54117-a230-4c10-b6fc-a906b592ddaf" ] +} +``` + +`properties` is the parameter array otherwise sent as the `properties` form field. `uploads` lists +the identifiers of complete staged uploads owned by the requesting user; each becomes an input file +under its staged filename, so a parameter such as `--zip` must name that file. `uploads` may be +empty, in which case the script is started without input files. + +The response is the same `202 Accepted` process representation as above, and the same status codes +apply. In addition: +* 404 Not found - if one of the uploads does not exist or belongs to another user +* 409 Conflict - if one of the uploads is incomplete, or is being or has been consumed by another request + +The script's own authorization is checked before the uploads are touched, so a refused request +leaves them resumable. Once the process exists its link is recorded on every consumed upload; +repeating the same request afterwards, including after a lost response, returns that same process +and never starts a second one. diff --git a/uploads.md b/uploads.md new file mode 100644 index 00000000..5fb109ef --- /dev/null +++ b/uploads.md @@ -0,0 +1,172 @@ +# Staged uploads + +[Back to the list of all defined endpoints](endpoints.md) + +Large files may be staged in bounded, sequential requests before another endpoint consumes them, +without TUS or a separate upload server. Staging creates nothing but the staged file: the resource +that eventually uses it (an import process, a workspace item, ...) is created by that resource's own +endpoint, which then applies its own authorization rules and returns its usual representation. +Currently [script invocation](scripts-endpoint.md#script-invocation-from-staged-uploads) consumes +staged uploads; the multipart forms of the consuming endpoints remain supported. + +All endpoints require an authenticated user. Each upload is bound to its creator; anyone else, +including administrators, receives `404` for it. Every request is authenticated on its own with the +normal JWT, and mutating requests also require the usual [CSRF protection](csrf-tokens.md). JWTs and +CSRF tokens are separate mechanisms; neither is replaced by the upload identifier. + +## Discovery and preflight + +The API root links to the resource: + +```json +{ + "_links": { + "uploads": { "href": "/api/core/uploads" } + } +} +``` + +**POST /api/core/uploads** + +Content type: `application/json`. Send only metadata, not the file body: + +```json +{ "name": "batch.zip", "size": 10485760 } +``` + +`size` is the exact complete file length in bytes. It must be positive and no larger than the +existing `spring.servlet.multipart.max-file-size` setting (DSpace default: `512MB`). A configured +unlimited value remains unlimited. The limit applies to the *whole file*, not to each chunk. +The filename must be nonblank, at most 255 characters, and contain no `..`, slash, backslash or +control characters. Consuming endpoints may apply stricter rules of their own when they use the file. + +The server validates the size **before reserving staging or receiving file bytes**. It returns +`413 Payload Too Large` for an oversized file. Clients must wait for a successful preflight before +sending any chunk, and show a file-size error immediately when rejected. + +Successful creation returns `201 Created`, a `Location` header and this HAL representation: + +```json +{ + "id": "18c54117-a230-4c10-b6fc-a906b592ddaf", + "name": "batch.zip", + "size": 10485760, + "chunkSize": 8388608, + "receivedBytes": 0, + "state": "UPLOADING", + "result": null, + "_links": { + "self": { "href": "/api/core/uploads/18c54117-a230-4c10-b6fc-a906b592ddaf" }, + "chunks": { + "href": "/api/core/uploads/18c54117-a230-4c10-b6fc-a906b592ddaf/chunks/{offset}", + "templated": true + } + } +} +``` + +The client must use the returned `chunkSize`, not assume the example value. It is the configured +`staged-upload.chunk-size` (default 8 MiB), capped by a finite `spring.servlet.multipart.max-request-size`. +Chunk requests are not multipart requests; the staging implementation enforces these limits itself. + +Creation can also return `400` for invalid metadata, `429` when the user's quota of active uploads +(`staged-upload.max-active-per-user`, default 8) is exhausted, or `409` when staging is briefly locked. +Creation is not idempotent: do not automatically repeat it after an uncertain response. An orphaned +reservation is cleaned up after the retention period. + +## Read committed progress + +**GET /api/core/uploads/<:uuid>** + +Returns `200` with the representation above. `receivedBytes` counts only fully committed chunks, +not an in-flight or partial chunk. Once a consuming endpoint has recorded what it created, `result` +holds that resource's link and a `result` link is added. Missing, cancelled, cleaned-up or +differently owned uploads return `404`. + +## Upload a chunk + +**PUT /api/core/uploads/<:uuid>/chunks/<:offset>** + +Content type: `application/octet-stream`. The body contains exactly the next +`min(chunkSize, size - offset)` bytes, without multipart encoding. The offset starts at zero, +is a multiple of `chunkSize`, and may not skip committed data. Send only one request at a time. + +Returns `200` with updated committed progress. A repeated PUT at an already committed offset +is idempotent **only when its bytes are identical**. A short body returns `400`, an oversized +body `413`, and a changed retry, invalid offset, concurrent write or an upload that is no longer +`UPLOADING` returns `409`. Partial data never advances committed progress. + +Each request is authenticated independently, so a JWT renewed between chunks is used by the next +chunk. A chunk can still fail on token expiry or a network error; retry only that chunk, or GET the +upload to recover its committed offset. Unauthorized users cannot enter the body-reading controller +method. An upstream proxy may nevertheless buffer a request before forwarding it; bounded chunks +limit the amount at risk and do not override deployment-specific proxy behavior. + +## Consume an upload + +A staged upload is consumed by the endpoint of the resource it is meant for, using a JSON body +that references the upload identifier instead of a multipart file, for example +[script invocation](scripts-endpoint.md#script-invocation-from-staged-uploads). Consumption is +only possible for a complete upload in state `UPLOADING`, owned by the requesting user; otherwise +the consuming endpoint returns `409` (or `404` for another user's upload). Several uploads can be +consumed by one request. + +The consuming endpoint records the link of the created resource on every upload it consumed **as +soon as that resource exists**, before anything that can still fail. Repeating the same request +afterwards, including after a lost HTTP response, returns that same resource and never creates a +second one; changed parameters in a repeated request do not modify it. If the resource was deleted +since, the response is `404`. + +State meanings: + +| State | Meaning | +| --- | --- | +| `UPLOADING` | Chunks may be uploaded; no consumption has begun. | +| `CONSUMING` | A consuming request has begun. | +| `CONSUMED` | Consumption succeeded and the staged data was removed; `result` links to what was created. | +| `FAILED` | Consumption failed. A known `result` still identifies the resource created before the failure. | + +There is a narrow crash window between creating the resource and publishing its receipt. A +`CONSUMING` or `FAILED` upload without a `result` is deliberately not retried automatically: +consuming it again returns `409`, and an administrator must inspect existing resources before +starting over. This is not a distributed transaction or an exactly-once guarantee across arbitrary +storage failure. + +## Cancel staging + +**DELETE /api/core/uploads/<:uuid>** + +Returns `204 No Content` after discarding an `UPLOADING` upload. Stop any in-flight transfer first. +Cancellation after consumption has begun returns `409`; it never deletes what was created. + +## Browser presentation + +The client owns upload percentage and approximate upload speed. The transfer rate is computed from +browser upload events over a short rolling window, includes retransmitted bytes and waiting time, +and declines to zero during a stall. Hide the rate after transfer or on interruption. + +Reserve 100% upload completion for acknowledgement of the final chunk, then show a distinct +"starting" phase while the consuming request runs. Upload completion is not completion of whatever +the consuming endpoint starts; that resource owns its own status. + +The reference client retains the selected File and receipt during navigation within the same +application instance. Browser reload, tab closure and restarting the browser lose that client +state; persistence or automatic cross-tab/cross-user resumption is not provided. + +## Deployment and cleanup + +`staged-upload.directory` defaults to `${dspace.dir}/var/staged-uploads`. It must be dedicated to +staging, writable only by the service account, and shared by backend nodes serving the same +repository. The filesystem must support Java file locks and atomic rename within a directory. +Deployment with independent per-node disks needs routing affinity instead; transparent failover is +not supported without shared staging. Do not expose this directory through a web server. + +`staged-upload.max-active-per-user` defaults to eight staging slots per user. Failed consumptions +retaining data continue to occupy a slot. `staged-upload.retention-hours` defaults to 24 hours; +hourly cleanup removes expired staging and receipts while skipping locked uploads. Accepted chunks +refresh retention; reading progress alone does not. Receipts are not permanent idempotency keys. + +Allow up to twice the file size while chunks are assembled, plus whatever storage the consuming +endpoint needs. Early size validation is not a reservation of physical disk space: disk failure, +quota exhaustion, network failure or invalid content can still prevent the consuming endpoint from +succeeding. Reverse proxies must allow the negotiated chunk size and suitable request timeouts.