Harden search validation and runtime safeguards

Author: afoote-mitre

.env.example

@@ -14,6 +14,7 @@ SearchPostFilterEvaluationChunkSize=250
 SearchOpenSearchRequestTimeoutMs=20000
 SearchRequestDeadlineMs=30000
 SearchReadinessTimeoutMs=5000
+SearchMaxResponseBytes=10485760
 RequestLogSuccessSampleRate=0.1
 RequestLogHealthChecks=false
 ShutdownTimeoutMs=35000

README.md

@@ -20,6 +20,7 @@ Copy `.env.example` to `.env` and replace or check the following values.
 - `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000`
 - `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000`
 - `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000`
+- `SearchMaxResponseBytes` - optional maximum serialized successful `/search` response size in bytes, default `10485760` (10 MiB)
 - `RequestLogSuccessSampleRate` - optional fraction of successful non-health-check access logs to retain, default `0.1`
 - `RequestLogHealthChecks` - optional flag to retain successful health-check access logs, default `false`
 - `ShutdownTimeoutMs` - optional graceful HTTP shutdown deadline, default `35000`
@@ -55,6 +56,7 @@ The following numeric bounds are enforced:
 | `SearchOpenSearchRequestTimeoutMs`        |                                                    1 through 300,000 |
 | `SearchRequestDeadlineMs`                 |                                                    1 through 600,000 |
 | `SearchReadinessTimeoutMs`                |                                                     1 through 60,000 |
+| `SearchMaxResponseBytes`                  |                                           65,536 through 104,857,600 |
 | `RequestLogSuccessSampleRate`             |                                                          0 through 1 |
 | `ShutdownTimeoutMs`                       |                                                    1 through 900,000 |
 | `HttpHeadersTimeoutMs`                    |                                                     1 through 60,000 |
@@ -159,7 +161,7 @@ String inputs are bounded before query construction:
 | ---------------------------- | ---------------------------------------------------------------------: |
 | `keywordSearch`              |                       512 characters and 64 whitespace-delimited terms |
 | `cveIds`                     | 4,096 characters overall, 100 IDs, and 4-19 sequence digits per CVE ID |
-| `cweId`                      |                                                          32 characters |
+| `cweId`                      |                      32 characters and `CWE-<positive integer>` format |
 | `cpeName`                    |                                                       2,048 characters |
 | `virtualMatchString`         |                                                       2,048 characters |
 | `versionStart`, `versionEnd` |                                                    256 characters each |
@@ -169,6 +171,8 @@ Requests that exceed a limit return HTTP 400 before contacting OpenSearch. These
 
 CVSS vector filters accept full or partial vectors containing at least one unique `KEY:VALUE` metric token. An optional vector header must identify a version supported by the selected parameter. Requested metric tokens can appear in any order, but all tokens must occur in the same stored vector string. CVSS and CWE filters search CNA-provided data only; ADP-provided enrichment is not currently included.
 
+The `cweId` filter accepts a positive decimal CWE identifier such as `CWE-79`. The prefix is case-insensitive, surrounding whitespace is trimmed, and valid input is normalized to uppercase before query construction. Zero, leading-zero identifiers such as `CWE-079`, missing or non-numeric identifiers, and suffixes return HTTP 400 without contacting OpenSearch.
+
 Publication and last-modified date filters support one-sided ranges. Values must use the exact UTC format `yyyy-MM-ddTHH:mm:ss.SSSZ`, including three fractional-second digits and the trailing `Z`. When both values are supplied, `pubStartDate` must be less than or equal to `pubEndDate`, and `lastModStartDate` must be less than or equal to `lastModEndDate`.
 
 ### Search Ordering and Pagination
@@ -179,6 +183,8 @@ Publication and last-modified date filters support one-sided ranges. Values must
 
 Pagination is limited to complete pages inside the first 10,000 results: `from + resultsPerPage` must not exceed 10,000. `totalPages` reports the smaller of the pages implied by the match count and the complete pages accessible inside that window. For example, a page size of 30 allows at most 333 pages. Standard-search totals are exact and are not reduced by this pagination calculation. Semantic CPE searches have a separate 10,000-candidate evaluation limit described below, so their totals can be incomplete. Point-in-time search and deep pagination remain deferred.
 
+Successful `/search` responses are also limited by serialized UTF-8 size. The default limit is 10 MiB and can be changed with `SearchMaxResponseBytes`. The API serializes the complete response and rechecks the overall search deadline before sending any bytes. If the successful payload exceeds the configured limit, the API returns HTTP 422 with `SEARCH_RESPONSE_TOO_LARGE`; reduce `resultsPerPage` and retry. Error responses bypass this success-payload limit so clients receive a complete error body. Match totals and the 500-record page-size limit are unchanged.
+
 Use `cveIds` to request up to 100 CVE records by comma-separated CVE ID. Each ID must contain the `CVE` prefix, a four-digit year, and a sequence number containing 4 to 19 digits. The prefix is accepted without regard to case and normalized to uppercase for the OpenSearch query.
 
 ```bash
@@ -321,7 +327,7 @@ An opt-in, read-only OpenSearch contract check is available for an approved envi
 npm run test:opensearch-contract
 ```
 
-Set `OpenSearchDomainEndpoint` and `OpenSearchCveIndex` in the environment or local `.env` before running it. `OpenSearchAllowUnknownSslCerts` remains optional and should be enabled only for local development against a self-signed certificate. The contract check sends a minimal readiness query, validates representative field capabilities, requires a concrete CVE record, and uses that record for a positive CVE-ID search before running the remaining bounded query shapes. It closes its OpenSearch client and emits sanitized JSON output. It never creates, updates, maps, or deletes indexes.
+Set `OpenSearchDomainEndpoint` and `OpenSearchCveIndex` in the environment or local `.env` before running it. `OpenSearchAllowUnknownSslCerts` remains optional and should be enabled only for local development against a self-signed certificate. The contract check sends a minimal readiness query, validates representative field capabilities, and requires a concrete CVE record with a structurally valid direct CNA CPE applicability entry. It uses that record for a positive CVE-ID search and one of its validated CPE criteria values for a positive candidate search before running the remaining bounded query shapes. It closes its OpenSearch client and emits sanitized JSON output. It never creates, updates, maps, or deletes indexes.
 
 The contract check is not part of public pull-request CI. The `OpenSearch Contract` GitHub Actions workflow runs it only through a manual dispatch against a selected protected GitHub environment. Configure that environment with:
 
@@ -331,13 +337,13 @@ The contract check is not part of public pull-request CI. The `OpenSearch Contra
 
 The workflow can run only from the trusted `dev` or `main` branch. The selected GitHub environment should additionally require reviewers and restrict deployment branches. OpenSearch secrets are exposed only to the final contract execution step, after checkout, Node setup, and dependency installation have completed.
 
-The selected runner must have network access to the endpoint. Change the workflow to an approved self-hosted runner before use when the index is not reachable from GitHub-hosted runners. The workflow performs readiness, field-capability, positive source-projection, exact-total, CVE ID keyword-subfield, keyword, publication-date, CVSS, and CPE candidate checks. Every search is read-only and returns at most one record. Connection values and credentials remain in environment secrets rather than source files or workflow text.
+The selected runner must have network access to the endpoint. Change the workflow to an approved self-hosted runner before use when the index is not reachable from GitHub-hosted runners. The workflow performs readiness, field-capability, positive CPE source-projection, exact-total, CVE ID keyword-subfield, keyword, publication-date, CVSS, and positive CPE candidate checks. Candidate responses are validated with the same CVE ID, applicability, node, CPE match, vulnerability, criteria, and version-range requirements used by application searches. Every search is read-only and returns at most one record. Connection values and credentials remain in environment secrets rather than source files or workflow text.
 
 The validation tiers have separate purposes:
 
 - Jest and `npm run quality` verify application behavior, API contracts, and regression coverage with controlled test doubles.
 - Pull-request container validation verifies production-image packaging, non-root execution, startup, liveness, and packaged API documentation.
-- The optional OpenSearch contract check verifies that the installed client can communicate with a non-empty externally managed index whose representative searchable fields satisfy the application contract.
+- The optional OpenSearch contract check verifies that the installed client can communicate with a non-empty externally managed index whose representative searchable fields and direct CNA CPE applicability records satisfy the application contract.
 - Load and performance testing remain in the separate automation performance-testing repository.
 
 Dependabot checks npm packages, Docker base-image digests, and GitHub Actions weekly. Version-update pull requests target the `dev` integration branch; GitHub security updates continue to follow the repository default-branch behavior. Both Dockerfiles pin the Node build image and Distroless runtime image to immutable multi-architecture digests; Dependabot updates those references through reviewable pull requests.
@@ -350,7 +356,9 @@ Access logs omit query strings and request bodies. By default, the API retains 1
 
 The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, shared-operation waiter saturation, queue wait timeout, search timeout, client disconnect, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, cancellation reason, and bounded queue or waiter state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included.
 
-Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, shared waiter counts, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, response bytes, JSON serialization time, process heap usage and per-request heap delta, cache entry and cached CVE ID counts, recent process-wide event-loop delay, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable.
+Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, shared waiter counts, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, response bytes, JSON serialization time, process heap usage and per-request heap delta, cache entry and cached CVE ID counts, recent process-wide event-loop delay, and total handler elapsed time. Oversized response failures are logged even when detailed timing is disabled and include the attempted byte count, configured maximum, and attempted serialization time. Leave detailed logging disabled during normal operation unless the deployment logging volume is acceptable.
+
+Candidate-discovery and page-fetch timings describe the underlying operation, which can be shared by multiple active requests. Every request that successfully waits for that operation reports the same operation timing while retaining its own cache status, in-flight status, waiter count, wait duration, deadline, response, and outcome. If the initiating client disconnects but another waiter remains, the surviving request still receives the completed operation metrics. Cache entries contain only candidate results, so a later cache hit does not replay historical candidate, post-filter, or queue timings; it reports only work performed for the current request, such as its page fetch.
 
 Event-loop delay uses one low-overhead process monitor sampled in rolling 10-second windows. Request events report the most recently completed window through `processEventLoopDelayMeanMs`, `processEventLoopDelayP99Ms`, `processEventLoopDelayMaxMs`, `processEventLoopDelayWindowMs`, and `processEventLoopDelaySampleAgeMs`. These values describe process-wide event-loop saturation near the request, not delay caused by that individual request. They are omitted until the first complete window is available, and concurrent requests read the same completed snapshot without resetting one another's measurements.
 
@@ -415,7 +423,7 @@ This helps most when multiple users submit the same expensive CPE search within
 
 The cache and cold-search queue are a temporary per-process proof of concept. They do not share state across ECS tasks, so each task warms and admits work independently depending on load balancer routing. Track cache-hit and in-flight-share rates, cold-search queue waits and rejections, cold-versus-warm latency, and OpenSearch time to decide their long-term value. If the benefit is low, remove this layer; if it is material, replace it with an AWS-managed shared cache and admission-control design.
 
-The request lifecycle and in-flight waiter registry are separate from the result cache, so a later shared-cache replacement does not need to change caller cancellation behavior. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, shared waiter limit, evaluation chunk size, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchInFlightMaxWaitersPerOperation`, `SearchPostFilterEvaluationChunkSize`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`.
+The request lifecycle and in-flight waiter registry are separate from the result cache, so a later shared-cache replacement does not need to change caller cancellation behavior. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, shared waiter limit, evaluation chunk size, OpenSearch timeout, overall request deadline, and successful response size can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchInFlightMaxWaitersPerOperation`, `SearchPostFilterEvaluationChunkSize`, `SearchOpenSearchRequestTimeoutMs`, `SearchRequestDeadlineMs`, and `SearchMaxResponseBytes`.
 
 ### HTTP Transport Timeouts