Skip to content

arbos/storage: shard the keccak cache to remove a global mutex - #4767

Closed
paranoiacblack wants to merge 5 commits into
OffchainLabs:masterfrom
paranoiacblack:sharded-arbos-hash-cache
Closed

paranoiacblack wants to merge 5 commits into
OffchainLabs:masterfrom
paranoiacblack:sharded-arbos-hash-cache

Conversation

@paranoiacblack

Copy link
Copy Markdown

Problem

Storage.cachedKeccak uses lru.NewCache[string, []byte], which guards one BasicLRU with a single sync.Mutex. Get takes it in write mode because a lookup reorders recency, so it is not a read-mostly lock.

Storage.mapAddress consults that cache once per storage slot, and every eth_call opens ArbOS state before executing an opcode:

BlockChainAPI.Call
  TransactionArgs.ToMessage                 transaction_args.go:525
    nodeinterface virtual_contracts.go:125
      OpenSystemArbosState -> OpenArbosState -> OpenL2PricingState -> OpenMultiGasFees
        Storage.NewSlot -> Storage.mapAddress -> Storage.cachedKeccak   storage.go:375
          common/lru.(*Cache).Get -> sync.(*Mutex).Lock                 lru.go:51

Block insertion opens ArbOS state through the same path, so RPC and block execution contend on one lock.

We hit this on a self-hosted archive node (v3.12.1, 64 vCPU) under eth_call load. Goroutine profile at the time:

count stack
10,629 total
3,247 rpc.(*handler).close -> WaitGroup.Wait
167 lru.(*Cache).Get -> Mutex.Lock
338 applyMessageWithEVM (actually executing)

The node held ~10% CPU with disk idle while block execution went from 3.4ms to 1,744ms p50 and it fell behind the chain. Adding cores did not help, which is what a single mutex looks like.

Change

Shard the cache across independently locked lru.Cache instances, selected by maphash of the key. Keys are a shared subspace prefix followed by a slot, so slicing a prefix would land a whole subspace on one shard; hashing avoids that.

storageKeyCacheSize becomes the per-shard capacity. With 64 shards the cache holds 65,536 entries rather than 1,024, roughly 6.5MB. That is deliberate: the existing "Hash cache full, we didn't expect that" warning fires on nodes serving real eth_call traffic. If you would rather hold total capacity constant, set storageKeyCacheSize to 16 and only the contention fix remains.

Benchmark

BenchmarkHashCacheGetParallel, 16-core arm64, -benchtime=3s. shards=1 is current behaviour.

goroutines shards=1 shards=8 shards=64
1 22.74 ns/op 21.37 ns/op 23.42 ns/op
8 119.2 ns/op 64.89 ns/op 27.40 ns/op
16 140.2 ns/op 85.16 ns/op 38.66 ns/op

Today's cache gets slower as goroutines are added (22.7 -> 140.2 ns). Sharded, it stays roughly flat. 3.6x at 16 goroutines, and the gap widens with core count.

Testing

go test passes for the new file: round-trip, power-of-two shard rounding, and key spread across shards.

I could not run the full arbos/storage suite locally because the package needs generated solgen bindings and I do not have the Solidity toolchain set up. The new code depends only on common/lru, hash/maphash and math/bits, and cachedKeccak is unchanged since Get/Add keep their signatures. Worth a CI run before trusting it.

Happy to adjust shard count, capacity, or split the capacity change into its own PR.

geth's lru.Cache guards one BasicLRU with a single sync.Mutex, and Get takes
it in write mode because a lookup reorders recency. Storage.mapAddress
consults that cache once per storage slot, and every eth_call opens ArbOS
state before executing an opcode, so the cache serialises RPC and block
execution process-wide.

Spread it over independently locked shards, selected by maphash of the key.
Keys are a shared subspace prefix followed by a slot, so slicing a prefix
would put a subspace on one shard; hashing avoids that.

storageKeyCacheSize becomes the per-shard capacity, so the cache now holds
64x as many entries (~6.5MB). That also addresses the "Hash cache full, we
didn't expect that" warning, which fires on nodes serving real eth_call load.
@CLAassistant

CLAassistant commented Oct 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@eljobe eljobe self-assigned this Oct 15, 2026

@eljobe eljobe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to add a changelog entry.

Comment thread arbos/storage/hashcache.go Resolved
Comment thread arbos/storage/hashcache_test.go Resolved
Comment thread arbos/storage/hashcache_test.go Resolved
@eljobe eljobe assigned paranoiacblack and unassigned eljobe Oct 16, 2026
paranoiacblack and others added 4 commits October 16, 2026 08:17
Co-authored-by: Pepper Lebeck-Jobe <eljobe@gmail.com>
Co-authored-by: Pepper Lebeck-Jobe <eljobe@gmail.com>
Co-authored-by: Pepper Lebeck-Jobe <eljobe@gmail.com>
The suggested TestShardedHashCacheSpreadsKeys body calls maphash.String
directly, which needs hash/maphash in the imports. Applying the suggestion
on its own left the package not compiling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@paranoiacblack

Copy link
Copy Markdown
Author

You also need to add a changelog entry.

Done, PTAL.

@paranoiacblack
paranoiacblack requested a review from eljobe October 16, 2026 14:30
@paranoiacblack paranoiacblack removed their assignment Oct 16, 2026
@eljobe eljobe self-assigned this Oct 28, 2026
@eljobe eljobe assigned joshuacolvin0 and unassigned eljobe Nov 9, 2026
@joshuacolvin0

Copy link
Copy Markdown
Member

Thanks, @paranoiacblack. Your diagnosis and profile were spot on and led directly to our fix.

We're closing this in favor of a different approach. The "Hash cache full" warning was a bug: ticket IDs and per-transaction slots were going into a cache meant only for keys fixed by code. Once those keys are kept out, the cache stays at about 50 entries and can be read with no lock. On 14 cores, opening ArbOS state drops from 24.7µs to 2.0µs.

It will ship in an upcoming release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants