Enable PIT semantic traversal by default

Author: afoote-mitre

.env.example

@@ -5,8 +5,8 @@ OpenSearchDomainEndpoint=
 SearchOpenSearchRequestTimeoutMs=20000
 SearchRequestDeadlineMs=30000
 SearchReadinessTimeoutMs=5000
-# Experimental PIT traversal; leave disabled until deployment validation is complete.
-SearchPitEnabled=false
+# PIT traversal is enabled by default; set false to restore bounded semantic searches.
+SearchPitEnabled=true
 SearchPitBatchSize=1000
 SearchPitKeepAliveMs=60000
 RequestLogSuccessSampleRate=0.1

README.md

@@ -38,7 +38,7 @@ Copy `.env.example` to an untracked `.env` and replace or check the following va
 - `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000`
 - `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000`
 - `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000`
-- `SearchPitEnabled` - experimental semantic PIT traversal, default `false`; leave disabled until deployment validation is complete
+- `SearchPitEnabled` - semantic PIT traversal, default `true`; set `false` to restore bounded semantic searches
 - `SearchPitBatchSize` - PIT candidate batch size, default `1000`; see [PIT Implementation Status](#pit-implementation-status)
 - `SearchPitKeepAliveMs` - PIT lifetime in milliseconds, default `60000`
 - `RequestLogSuccessSampleRate` - optional fraction of successful access logs to retain, default `0.1`; includes health checks when they are enabled for logging
@@ -342,7 +342,7 @@ npm run coverage
 npm run quality
 ```
 
-An importable Postman collection for manual `/search` contract and filter testing is documented in [test/postman/README.md](test/postman/README.md). Set its `expectedSearchPitEnabled` assertion variable to match the server's PIT setting; it defaults to `false` and does not enable PIT. The complete collection targets a fully populated index like those used by shared managed environments; data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly.
+An importable Postman collection for manual `/search` contract and filter testing is documented in [test/postman/README.md](test/postman/README.md). Set its `expectedSearchPitEnabled` assertion variable to match the server's PIT setting; it defaults to `true` and does not enable PIT. The complete collection targets a fully populated index like those used by shared managed environments; data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly.
 
 ## Operational Logging
 
@@ -356,16 +356,16 @@ Deadline checks also compare elapsed monotonic time, so they do not depend on th
 
 ## Semantic Post-Filtering
 
-With PIT disabled (the default), two separate 10,000-record controls apply to `/search`:
+Two separate 10,000-record controls apply to `/search`, depending on the server's PIT setting:
 
 - Every requested page must fall within the first 10,000 results. Direct OpenSearch filters still report an exact total that can exceed 10,000, but pages beyond that window are rejected.
-- Filters that require semantic post-filtering inspect at most 10,000 search candidates. Their reported total and pagination describe only matches found within that candidate set.
+- Only when PIT is disabled, filters that require semantic post-filtering inspect at most 10,000 search candidates. Their reported total and pagination describe only matches found within that candidate set. PIT is enabled by default and removes this candidate cap, not the public pagination limit.
 
-CVSS vector filters, SSVC filters, date-bounded positive `knownExploited` searches, `cpeName`, and `virtualMatchString` searches use semantic post-filtering where OpenSearch cannot guarantee the required same-entry or version-comparison behavior. With PIT disabled, these searches retrieve up to 10,000 lightweight candidates, apply semantic checks in Node.js, paginate the matching CVE IDs, and fetch the requested page as complete CVE records. Every slash-delimited token supplied to one CVSS vector filter must match a complete token in the same eligible vector; a base token such as `AV:N` does not match a modified token such as `MAV:N`.
+CVSS vector filters, SSVC filters, date-bounded positive `knownExploited` searches, `cpeName`, and `virtualMatchString` searches use semantic post-filtering where OpenSearch cannot guarantee the required same-entry or version-comparison behavior. These searches retrieve lightweight candidates, apply semantic checks in Node.js, paginate the matching CVE IDs, and fetch the requested page as complete CVE records. Every slash-delimited token supplied to one CVSS vector filter must match a complete token in the same eligible vector; a base token such as `AV:N` does not match a modified token such as `MAV:N`.
 
-In the default mode, the reported total and pagination are complete only within that bounded candidate set; qualifying records beyond the first 10,000 search candidates may not be represented. When a semantic search reaches that limit, the response sets `pagination.totalIsExact` to `false` and includes a `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Semantic searches that remain below the limit and direct searches set `totalIsExact` to `true`. A `knownExploited` filter without date boundaries uses a direct query and retains ordinary exact-total behavior.
+By default, PIT traversal evaluates successive candidate batches to completion rather than stopping at 10,000 candidates. Successful responses have exact totals and no `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Public pages must still fit entirely within the first 10,000 matching results. `totalPages` is calculated from the full total and may exceed the number of accessible pages; it does not override the pagination window. PIT is server configuration, not a request parameter. No public cursor is returned. See the deployment prerequisites below.
 
-With experimental PIT traversal enabled, semantic searches evaluate successive candidate batches to completion rather than stopping at 10,000 candidates. Successful responses have exact totals and no `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Public pages must still fit entirely within the first 10,000 matching results. `totalPages` is calculated from the full total and may exceed the number of accessible pages; it does not override the pagination window. PIT is server configuration, not a request parameter. No public cursor is returned. See the rollout prerequisites below before enabling this path.
+With PIT disabled using `SearchPitEnabled=false`, the reported total and pagination are complete only within the bounded candidate set; qualifying records beyond the first 10,000 search candidates may not be represented. When a semantic search reaches that limit, the response sets `pagination.totalIsExact` to `false` and includes a `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Semantic searches that remain below the limit and direct searches set `totalIsExact` to `true`. A `knownExploited` filter without date boundaries uses a direct query and retains ordinary exact-total behavior.
 
 `/search` orders search results by CVE ID so repeated shallow-page requests use a deterministic sort. The response `pagination.pageSize` preserves the requested `resultsPerPage` value, while `pagination.resultsReturned` reports the number of records actually returned and may be smaller for a partial or out-of-range page. PIT provides a consistent snapshot within one semantic request, including its full-record retrieval. Each subsequent HTTP request opens a new PIT, so index updates between separate page requests can still shift results. Direct searches do not use PIT.
 
@@ -375,9 +375,9 @@ OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TI
 
 ### PIT Implementation Status
 
-Internal point-in-time (PIT) traversal is implemented for semantic `/search` requests when `SearchPitEnabled=true`, including deadline enforcement, completion/error validation, Swagger descriptions, and mode-aware Postman assertions. It remains disabled by default and is not validated in a managed deployment; leave it disabled until the rollout checks below are complete. Disabled deployments retain the existing bounded semantic path. Direct `/search` queries, the public pagination window, readiness behavior, and `/webSearch` are unchanged.
+Internal point-in-time (PIT) traversal is enabled by default for semantic `/search` requests, including deadline enforcement, completion/error validation, Swagger descriptions, and mode-aware Postman assertions. Explicit `SearchPitEnabled=false` overrides remain honored and retain the existing bounded semantic path. Deployments adopting the new default must support PIT; unsupported or unauthorized operations return errors without automatic fallback. Direct `/search` queries, the public pagination window, readiness behavior, and `/webSearch` are unchanged.
 
-Local validation against OpenSearch 3.5.0 covered traversal beyond 10,000 candidates, exact totals and page retrieval, snapshot consistency during index changes, and controlled timeout, cancellation, cleanup, and incomplete-response cases. The Postman collection passed with PIT disabled and enabled, including against the built runtime container. These checks establish local functionality, not production capacity, deployed permissions, or behavior during managed-service restarts and deployments.
+Local validation against OpenSearch 3.5.0 covered traversal beyond 10,000 candidates, exact totals and page retrieval, snapshot consistency during index changes, and controlled timeout, cancellation, cleanup, and incomplete-response cases. The Postman collection passed with PIT disabled and enabled, including against the built runtime container. Separate minimal create/search/delete probes confirmed PIT access through deployed application clients in the managed environments. These checks do not establish production capacity, complete managed traversal, or behavior during managed-service restarts and deployments.
 
 The provider can create a PIT on the configured index, search through it without specifying an index again, and delete that specific context. Creation rejects partial shard snapshots. Every PIT search, including an empty final batch, must account for all shards successfully and report no timeout, early termination, or shard failures. Search shards skipped because they cannot match are allowed and are already included in the successful shard count; skipped creation shards are not allowed. Hit-shape, candidate-order, and full-page checks apply as well. The managed lifecycle helper attempts cleanup after success, failure, or cancellation with an independent timeout of at most five seconds. Cleanup failures emit a fixed warning without PIT IDs or upstream error details and do not replace the operation's result or error. If canceled creation later delivers a PIT ID, cleanup is attempted then; contexts whose IDs never reach the application must expire through their keep-alive period.
 
@@ -391,19 +391,19 @@ Complete page records are retrieved through the same PIT, restored to the select
 
 The configuration follows the existing `AppConfig` environment overrides:
 
-- `SearchPitEnabled` maps to `features.searchPitEnabled` and defaults to `false`. Unrecognized boolean strings leave PIT disabled.
+- `SearchPitEnabled` maps to `features.searchPitEnabled` and defaults to `true`. Explicit `false` overrides restore the bounded path. Unrecognized boolean strings leave PIT disabled. Existing environment overrides take precedence over the new default; configuration changes require an API restart or redeployment.
 - `SearchPitBatchSize` maps to `search.pitBatchSize` and accepts integers from 1 through 10,000, defaulting to 1,000. This is a per-batch size, not a total candidate limit.
 - `SearchPitKeepAliveMs` maps to `search.pitKeepAliveMs` and accepts positive safe integers, defaulting to 60,000 milliseconds. Each PIT search refreshes its lifetime. This does not extend the overall request deadline and must fit the deployed cluster's PIT lifetime limits.
 
 Missing or out-of-range numeric settings fall back to their defaults. The initial batch size and lifetime are starting points for validation, not established production capacity settings.
 
-Before enabling a completed PIT implementation, verify these deployment prerequisites:
+Before deploying with PIT enabled, verify these deployment prerequisites:
 
 1. The actual domain version supports PIT. Amazon OpenSearch Service supports PIT on OpenSearch 2.5 and later; the JavaScript client version is not the server version.
 2. The API's effective OpenSearch identity has index-scoped PIT create, search, and delete permissions, together with any required domain-policy permissions. An AWS administrator CLI session or a successful `/api/ready-check` does not establish these permissions for the application.
 3. The configured index or alias supports a complete, unique sort order. Current queries sort by `cveMetadata.cveId.keyword`, and readiness checks its searchable/aggregatable keyword capability, not uniqueness. Confirm one searchable document per CVE ID across the resolved indices, with keyword sort values matching source CVE IDs. If that is not the case, revise the sort and continuation validation to use an existing supported unique tiebreaker before enabling traversal. Do not assume a tiebreaker supported by another engine or version is available.
 4. Each index supports the chosen per-batch size, and the deployment has headroom for retained PIT segments and open contexts. Verify effective context and lifetime limits with the infrastructure owner; do not assume managed-service settings can be changed.
-5. PIT lifecycle, traversal, cancellation, cleanup, and failure handling have been validated against the deployed version and application identity. The helpers use the installed client's `createPit()` and `deletePit()` methods. Local runtime validation has passed, but verification in the intended managed environment remains pending. The [Postman PIT-mode guidance](test/postman/README.md#pit-modes) distinguishes response checks from lifecycle and capacity validation. No cache, queue, or telemetry subsystem is being reintroduced.
+5. PIT lifecycle, traversal, cancellation, cleanup, and failure handling have been validated against the deployed version and application identity. The helpers use the installed client's `createPit()` and `deletePit()` methods. Local runtime validation and minimal managed PIT access probes have passed; complete managed traversal, failure handling, and capacity validation remain pending. The [Postman PIT-mode guidance](test/postman/README.md#pit-modes) distinguishes response checks from lifecycle and capacity validation. No cache, queue, or telemetry subsystem is being reintroduced.
 
 Rollback is a server configuration change: set `SearchPitEnabled=false` and restart or redeploy the API so new requests use the bounded path again. This restores the 10,000-candidate limitation and its warnings; it does not preserve PIT totals or snapshots. Existing requests must be allowed to finish cleanup or be canceled, and contexts whose cleanup could not complete expire according to their keep-alive. No index or mapping migration is involved.