Set default memory resources for the ZooKeeper quorum

[delthas]

What does this PR do, and why do we need it?

Sets defaults.zookeeperResources in solution/zenkoversion.yaml, alongside the existing defaults.kafkaResources. Without it the quorum pods run BestEffort and the zookeeper image sizes its JVM heap as a percentage of the host memory instead of the container limit (RD-1962 finding).

Which issue does this PR fix?

Fixes ZENKO-5316.

Special notes for your reviewers:

Requires the zenko-operator supporting the field (scality/zenko-operator#624). With an older operator the unknown field is pruned by the CRD, so merging this first is harmless.

Issue: ZENKO-5316

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[francoisferrand]

zookeeper is used for kafka quorum, but also internally for many things in backbeat (depending on the features) : as cache, status....
→ are we sure about these limits?

[delthas]

Fair challenge — we grounded the value with data rather than picking it by feel. Consolidated analysis:

Method

Scality sosreports embed a 24h hourly snapshot of the platform's Prometheus (sos_commands/metalk8s/metrics/, one JSON file per metric). We extracted ZooKeeper memory from:

• the three RD-1962 sosreports (4.2.2 + 4.3.0-rc.1 AWS pair, plus the 4.3.0 HW lab under perf load), and
• the fleet sosreport inventory: ~240 distinct clusters (customers + QA, ARTESCA 2.x → 4.3, deduped by installation id, most recent report per cluster).

Per quorum pod: jvm_memory_bytes_committed/max{area="heap"} (what the heap flags resolved to), jvm_memory_pool_bytes_used{pool="G1 Old Gen"} (post-collection survivors ≈ live set), container RSS, znode count / data size.

What the data shows

• Where the problem comes from: zenko 2.13.9 swapped pravega/zookeeper:0.2.15 (stock fixed ~1 GB -Xmx) for the adobe 3.8.4 image whose zkEnv.sh sets -XX:InitialRAMPercentage=30 -XX:MaxRAMPercentage=70 (ZENKO-5198 — originally a CI/JDK fix). Without a cgroup limit, those percentages resolve against host RAM. The fleet split is perfectly clean: every cluster on ≤ 2.13.5-1 commits ~1 GB flat; every cluster on ≥ 2.13.9 commits 30% of node RAM — up to 315 GB committed / 736 GB max heap on 1 TB nodes, worst observed pod at 149.8 GB RSS, for a coordination service holding ~64 KB of znodes.
• What ZK actually keeps alive: G1 Old Gen (survivors after collection) never exceeds 15 MB on any cluster, including the HW lab under bench load. GC works fine — the multi-GiB "heap used" readings are young-gen garbage piling up because a huge Eden rarely triggers a collection.
• The proven baseline: 207 clusters ran for years on the old fixed ~1 GB heap, zero known ZK OOMs. Per-pod RSS there: median 0.4 GB, p99 0.6 GB, all-time max 1.1 GB. Note that max is just the 1 GB heap fully touched plus JVM overhead — under any fixed heap, RSS converges to ~Xmx over uptime regardless of need, so observed RSS is an artifact of the old Xmx, not evidence of demand.

Why 1Gi, and why it's safe

The honest sizing anchor is live set + burst headroom (request buffers: jute.maxbuffer 1 MB default × ~20 observed connections, snapshot serialization, watch fires — tens of MB at fleet scale), not observed RSS.

The image's RAMPercentage flags are container-aware, so with a 1Gi limit the same zkEnv line resolves to a ~717 MB max heap:

• ≈ 48× the worst live set ever observed on any cluster;
• above everything the bounded fleet ever actually used (median pod touched 0.4 GB over years of production);
• ~300 MB of the limit left for non-heap (observed metaspace + buffers: ~40 MB);
• failure mode of undersizing is merely more frequent sub-millisecond young GCs — three orders of magnitude below ZK session timeouts;
• requests == limits makes the pods Guaranteed QoS and the quorum footprint visible to the scheduler (the original RD-1962 crash was a scheduling-headroom problem).

[francoisferrand]

is this really the issue we had in production? If Zookeeper were actually "easting" the memory, it would have affected the system already (i.e. prevented others from using memory); or is there something else causing this to happen now ?

[delthas]

Requested @DarkIsDude in place of Maha Benzekri, who is currently on PTO.

[francoisferrand]

Where the problem comes from: zenko 2.13.9 swapped pravega/zookeeper:0.2.15 (stock fixed ~1 GB -Xmx) for the adobe 3.8.4 image whose zkEnv.sh sets -XX:InitialRAMPercentage=30 -XX:MaxRAMPercentage=70

this is the information which was missing! thanks!