Today caching is memory only. That caps how much hot data we can keep close to the node, and cold reads go straight to object storage (higher latency and limited by S3 RPS for massive fanout).
Add a disk tier (EBS volume or local NVMe) that sits between memory and S3:
- Write through (Optional): data lands on disk on the way to object storage, so recent data is readable locally without an S3 round trip
- Read through (A need): object fetches from S3 get cached on disk, so repeated reads of the same range are cheap
- Memory stays the first tier, disk is the second, S3 is the source of truth
Things to figure out:
- Caching and Eviction policy and a size cap for the disk tier
- How this interacts with the existing block cache
Today caching is memory only. That caps how much hot data we can keep close to the node, and cold reads go straight to object storage (higher latency and limited by S3 RPS for massive fanout).
Add a disk tier (EBS volume or local NVMe) that sits between memory and S3:
Things to figure out: