Fix cooldown and denylist gaps

docs/configuration.md

@@ -1232,15 +1232,17 @@ Maven caveats:
 
 Helm uses each release's `created` time from the repository's `index.yaml`. Releases in cooldown are left out of the index the proxy serves, and their archives aren't served; a release without `created` is let through. Charts pulled over OCI have no publish time and aren't held back.
 
-Cargo, RubyGems, Hex, pub.dev, Conda and Helm check artifact downloads against cooldown, including files cached before the policy changed. When a publish-time lookup fails, an already-cached file is still served; an uncached download fails with 502, or 503 with `Retry-After` when the rate-limit gate holds the request. Conda checks timestamps per file because builds of one version can have different publish times.
+Cargo, RubyGems, Hex, pub.dev, Conda and Helm check artifact downloads against cooldown, including files cached before the policy changed. When a publish-time lookup fails, an already-cached file is still served; an uncached download fails with 502, or 503 with `Retry-After` when the rate-limit gate holds the request. Conda checks timestamps per file because builds of one version can have different publish times. As in conda, a repodata `timestamp` is read as milliseconds when it is above 253402300799 (the end of year 9999 in seconds) and as seconds otherwise, since some older entries use seconds.
 
 Conda repodata and Helm indexes are parsed once per cached document for download checks. With `cache_metadata` off, the parsed index is kept for one minute, including the fetch, so an install can reuse it across files. Conda streams repodata when its compressed copy exceeds `metadata_max_size`, under the same decompressed size bound as the metadata filter.
 
 RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api`) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered.
 
 For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata`](#metadata-caching) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified`. Publish times are compared with the current time on each request, and the filtered result is kept until the first version it held back leaves cooldown (see [rewritten metadata cache](#rewritten-metadata-cache)); a newly published version shows up within `metadata_ttl`, as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed.
 
-With cooldown on, the proxy asks npm for full packuments, since the abbreviated ones it uses otherwise carry no publish times. Abbreviated packuments cached before cooldown was turned on are fetched again in full rather than served, including as a stale copy when the upstream is down. When `latest` is in cooldown, the packument tags the newest version out of cooldown instead, and `npm dist-tag ls` shows the same tags.
+With cooldown on, the proxy asks npm for full packuments, since the abbreviated ones it uses otherwise carry no publish times. Abbreviated packuments cached before cooldown was turned on are fetched again in full rather than served, including as a stale copy when the upstream is down. When `latest` is in cooldown, the packument tags the newest version out of cooldown instead; other tags that name a version in cooldown are removed, as for the denylist, and `npm dist-tag ls` shows the same tags. A tarball request for a version the cached packument doesn't list yet fetches the packument again, within `metadata_ttl` or not, and is decided on what the upstream lists now; if the packument can't be fetched, the download is not let through, as when a publish time can't be read.
+
+In pub.dev package metadata, when `latest` is in cooldown it names the newest stable version out of cooldown instead, or the newest prerelease when no stable version is left, and it is left out when every version is in cooldown.
 
 Conda cooldown covers `repodata.json`, `current_repodata.json`, `repodata_from_packages.json` and sharded repodata (CEP 16). Other forms of the repodata, such as `repodata.json.zst`, `repodata.json.bz2` and `repodata.jlap` patches, would list versions still in cooldown, so with cooldown on they are answered 404 and conda and mamba fall back to `repodata.json`. With `cache_metadata`, `repodata.json` is read through the metadata cache as it is without cooldown: within `metadata_ttl` the stored copy is filtered as it is read from storage, after it the upstream is asked with its ETag, and while the upstream is down the stale copy is filtered. Once a document has been filtered, the proxy remembers the ETag of the result until the first entry it left out leaves cooldown, so a client revalidating with `If-None-Match` gets `304 Not Modified` without the document being filtered or sent again. The first response for each document has no ETag, as it is sent while it is filtered. Without `cache_metadata`, `repodata.json` is fetched and filtered on every request. Clients cache shards by the hash in the shard index, so with cooldown on the proxy serves a rewritten index whose shard hashes change every hour and filters each shard as it is fetched. Versions leaving cooldown show up within the hour, and clients fetch the shards they need again once an hour. If a channel's shard index points packages or shards somewhere the proxy doesn't route, the proxy answers 404 for it and clients fall back to `repodata.json`.
 
@@ -1268,6 +1270,13 @@ Names and versions are compared the way each ecosystem treats them:
 - PyPI names ignore case, and runs of `-`, `_` and `.` are equivalent (PEP 503),
   so `pkg:pypi/zope.interface@6.0` also covers `zope_interface-6.0-*.whl`.
 - NuGet ids and versions ignore case.
+- Debian, RPM, Alpine, Arch and FreeBSD entries ignore the distribution
+  namespace: `pkg:deb/debian/curl@8.5.0-2` and `pkg:deb/curl@8.5.0-2` are the
+  same entry.
+- Debian and RPM filenames leave out the epoch, so an entry with one also
+  covers the version without it: `pkg:deb/foo@1:2.3-1` denies
+  `foo_2.3-1_amd64.deb`. An entry without an epoch only matches a version
+  without one.
 - A RubyGems entry also covers that version's platform gems:
   `pkg:gem/nokogiri@1.16.0` denies `nokogiri-1.16.0-x86_64-linux.gem`.
 - Composer versions ignore a leading `v`, which Packagist keeps from the tag:
@@ -1296,7 +1305,7 @@ Names and versions are compared the way each ecosystem treats them:
   channel only.
 
 Denied versions are removed from npm packuments, Cargo sparse indexes, and PyPI
-simple HTML/JSON and release metadata, including responses served from cached
+simple HTML/JSON (files and the JSON `versions` list) and release metadata, including responses served from cached
 metadata during an upstream outage. No publication timestamp or enabled cooldown
 is required. npm tags pointing at denied versions are removed; `latest` is moved
 to the highest remaining stable version when available. Other tags are not

internal/denylist/denylist.go

@@ -40,6 +40,13 @@ func New(packages []string) (*Policy, error) {
 			p.packages[key] = make(map[string]bool)
 		}
 		p.packages[key][id.version] = true
+		// Debian and RPM filenames leave the epoch out, so an entry with
+		// one also matches the version without it.
+		if pkg.Type == "deb" || pkg.Type == "rpm" {
+			if _, version, ok := strings.Cut(id.version, ":"); ok && version != "" {
+				p.packages[key][version] = true
+			}
+		}
 	}
 	return p, nil
 }
@@ -128,6 +135,9 @@ func genericSwiftRegistry(pkg *purl.PURL) (scope, name string, ok bool) {
 
 // identify compares PURLs in purl's comparison form. The generic
 // swift-registry spelling is the proxy's own, so it is mapped here first.
+// The namespace of a distro type names the distribution, which handlers
+// include for some types and not others, so it is dropped as retention and
+// the inventory do.
 func identify(pkg *purl.PURL) identity {
 	if scope, name, ok := genericSwiftRegistry(pkg); ok {
 		pkg = purl.New("swift", scope, name, pkg.Version, nil)
@@ -138,6 +148,10 @@ func identify(pkg *purl.PURL) identity {
 		// An entry without a channel matches every channel.
 		channel = c.Qualifier("channel")
 	}
-	key := purl.New(c.Type, c.Namespace, c.Name, "", nil).String()
+	namespace := c.Namespace
+	if packageurl.DistroType(c.Type) {
+		namespace = ""
+	}
+	key := purl.New(c.Type, namespace, c.Name, "", nil).String()
 	return identity{key: packageurl.PolicyPackageKey(key), channel: channel, version: c.Version}
 }

internal/denylist/denylist_test.go

@@ -90,3 +90,38 @@ func TestEcosystemSpellings(t *testing.T) {
 		t.Error("pypi versions not found under another spelling")
 	}
 }
+
+// TestDistroNamespaces checks that the distribution namespace of a distro
+// package is ignored on both sides, as handlers build some of these PURLs
+// without one and some with one.
+func TestDistroNamespaces(t *testing.T) {
+	p, err := New([]string{"pkg:deb/debian/curl@8.5.0-2", "pkg:rpm/fedora/curl@8.6.0-1.fc40", "pkg:apk/busybox@1.36.1-r15", "pkg:alpm/glibc@2.39-1"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, value := range []string{"pkg:deb/curl@8.5.0-2", "pkg:rpm/curl@8.6.0-1.fc40", "pkg:apk/alpine/busybox@1.36.1-r15", "pkg:alpm/arch/glibc@2.39-1"} {
+		if !p.Denied(value) {
+			t.Errorf("not denied: %s", value)
+		}
+	}
+	if p.Denied("pkg:deb/curl@8.5.0-3") {
+		t.Error("other version denied")
+	}
+}
+
+// TestDistroEpochs checks that a deb or rpm entry with an epoch matches the
+// version as it appears in a filename, which leaves the epoch out.
+func TestDistroEpochs(t *testing.T) {
+	p, err := New([]string{"pkg:deb/foo@1:2.3-1", "pkg:rpm/bar@2:1.0-1.fc40"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, value := range []string{"pkg:deb/foo@2.3-1", "pkg:deb/foo@1:2.3-1", "pkg:rpm/bar@1.0-1.fc40"} {
+		if !p.Denied(value) {
+			t.Errorf("not denied: %s", value)
+		}
+	}
+	if p.Denied("pkg:deb/foo@2.3-2") {
+		t.Error("other version denied")
+	}
+}

internal/handler/conda.go

@@ -432,8 +432,18 @@ func (c *countingWriter) Write(p []byte) (int, error) {
 	return n, err
 }
 
-// condaTimestampDivisor converts Conda's millisecond timestamps to seconds.
-const condaTimestampDivisor = 1000
+// condaMaxSecondsTimestamp is the last second of year 9999. Like conda, a
+// larger repodata timestamp is read as milliseconds and a smaller one as
+// seconds, which some older entries use.
+const condaMaxSecondsTimestamp = 253402300799
+
+// condaTime returns the time a repodata timestamp names.
+func condaTime(timestamp float64) time.Time {
+	if timestamp > condaMaxSecondsTimestamp {
+		return time.UnixMilli(int64(timestamp)).Truncate(time.Second)
+	}
+	return time.Unix(int64(timestamp), 0)
+}
 
 const (
 	repodataWriteBuffer = 64 << 10
@@ -503,7 +513,7 @@ func (f *repodataFilter) inCooldown(filename string, raw []byte) bool {
 	if entry.Name == "" || entry.Timestamp == 0 {
 		return false
 	}
-	publishedAt := time.Unix(int64(entry.Timestamp)/condaTimestampDivisor, 0)
+	publishedAt := condaTime(entry.Timestamp)
 	decision := f.h.proxy.cooldown().Evaluate("conda", canonicalPackagePURL("conda", entry.Name), publishedAt, f.now)
 	if decision.Allowed {
 		return false
@@ -520,14 +530,14 @@ func (f *repodataFilter) inCooldown(filename string, raw []byte) bool {
 	return true
 }
 
-// recordInCooldown reports whether a package record with a millisecond
+// recordInCooldown reports whether a package record with a repodata
 // timestamp was published too recently. Records without a name or
 // timestamp are kept.
 func (h *CondaHandler) recordInCooldown(filename, name, version string, timestamp float64) bool {
 	if name == "" || timestamp == 0 {
 		return false
 	}
-	publishedAt := time.Unix(int64(timestamp)/condaTimestampDivisor, 0)
+	publishedAt := condaTime(timestamp)
 	if h.proxy.cooldown().IsAllowed("conda", canonicalPackagePURL("conda", name), publishedAt) {
 		return false
 	}
@@ -620,7 +630,7 @@ func readRepodataIndex(r io.Reader) (map[string]time.Time, error) {
 			}
 			var e repodataEntry
 			if json.Unmarshal(raw, &e) == nil && e.Timestamp != 0 {
-				index[entry.Key] = time.Unix(int64(e.Timestamp)/condaTimestampDivisor, 0)
+				index[entry.Key] = condaTime(e.Timestamp)
 			}
 			return nil
 		})

internal/handler/conda_shards.go

@@ -378,7 +378,8 @@ func rewriteCondaShardIndex(body []byte, indexURL string, mask [condaShardHashLe
 }
 
 // condaShardRecord holds the fields cooldown needs from a shard record.
-// Timestamps are milliseconds, encoded as an integer or a float.
+// Timestamps are an integer or a float, in seconds or milliseconds as
+// condaTime reads them.
 type condaShardRecord struct {
 	Name      string `msgpack:"name"`
 	Version   string `msgpack:"version"`

internal/handler/conda_test.go

@@ -3,9 +3,13 @@ package handler
 import (
 	"bytes"
 	"encoding/json"
+	"fmt"
 	"log/slog"
+	"maps"
 	"net/http"
 	"net/http/httptest"
+	"slices"
+	"strings"
 	"testing"
 	"time"
 
@@ -133,6 +137,45 @@ func TestCondaCooldownFiltering(t *testing.T) {
 	}
 }
 
+// TestCondaCooldownSecondTimestamps checks that a timestamp in seconds,
+// which some repodata entries carry, is read as seconds, as conda does, in
+// both the filter and the download index.
+func TestCondaCooldownSecondTimestamps(t *testing.T) {
+	recent := time.Now().Add(-time.Hour)
+	body := fmt.Sprintf(`{"packages":{"seconds-1.0-0.tar.bz2":{"name":"seconds","version":"1.0","timestamp":%d},`+
+		`"millis-1.0-0.tar.bz2":{"name":"millis","version":"1.0","timestamp":%d}}}`, recent.Unix(), recent.UnixMilli())
+
+	proxy := testProxy()
+	proxy.Cooldown = &cooldown.Config{Default: "3d"}
+	h := &CondaHandler{proxy: proxy, proxyURL: "http://localhost:8080"}
+	filtered, err := filterRepodataBytes(h, []byte(body))
+	if err != nil {
+		t.Fatal(err)
+	}
+	var result struct {
+		Packages map[string]json.RawMessage `json:"packages"`
+	}
+	if err := json.Unmarshal(filtered, &result); err != nil {
+		t.Fatal(err)
+	}
+	if len(result.Packages) != 0 {
+		t.Errorf("packages = %v, want both left out", slices.Sorted(maps.Keys(result.Packages)))
+	}
+
+	index, err := readRepodataIndex(strings.NewReader(body))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(index) != 2 {
+		t.Fatalf("index has %d entries, want 2", len(index))
+	}
+	for filename, published := range index {
+		if published.Unix() != recent.Unix() {
+			t.Errorf("%s published at %v, want %v", filename, published, recent)
+		}
+	}
+}
+
 func TestCondaCooldownFilteringWithPackageOverride(t *testing.T) {
 	now := time.Now()
 	recentTimestamp := float64(now.Add(-2 * time.Hour).UnixMilli())

internal/handler/handler.go

@@ -1829,7 +1829,7 @@ func (p *Proxy) staleMetadata(ctx context.Context, ecosystem, cacheKey string, v
 
 	p.Logger.Info("serving metadata from cache",
 		"ecosystem", ecosystem, "key", cacheKey)
-	return metadataResult{body: data, contentType: ct, contentEncoding: entry.ContentEncoding.String, stale: true}.describedBy(entry)
+	return metadataResult{body: data, contentType: ct, contentEncoding: entry.ContentEncoding.String, stale: true, upstreamErr: err}.describedBy(entry)
 }
 
 // metadataResult is what a metadata lookup hands back. A shared fetch gives
@@ -1843,8 +1843,9 @@ type metadataResult struct {
 	digest string
 	err    error
 	// stale is set when the body is the cached copy served because upstream
-	// failed.
-	stale bool
+	// failed, and upstreamErr is that failure.
+	stale       bool
+	upstreamErr error
 	// fromCache is set when the body is a stored copy within its TTL, read
 	// without asking upstream.
 	fromCache bool