buildkit: let a NodePool span instance sizes, add m6id.12xlarge to prod amd64 - #1007
Open
huydhn wants to merge 1 commit into
Open
buildkit: let a NodePool span instance sizes, add m6id.12xlarge to prod amd64#1007huydhn wants to merge 1 commit into
huydhn wants to merge 1 commit into
Conversation
Capacity reportcommit ✅ simulate-cluster✅ analyze-utilization |
tofu plan — meta-prod-aws-ue2✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-uw1✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
huydhn
marked this pull request as ready for review
August 13, 2026 01:24
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
huydhn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
The amd64 pool was pinned to one instance type, on-demand only, so
Karpenter had no fallback. When us-east-2 ran out of m6id.24xlarge
capacity it logged "nodepool requirements filtered out all instance
types" against ~63 ICE rejections in 13 minutes across all three AZs,
stalling at 12 nodes while KEDA asked for 37 replicas.
m6id.12xlarge is half a 24xlarge, so it holds one 42 vCPU / 155Gi pod
where the 24xlarge holds two — same pod spec, same $/pod-hour, same 88%
packing.
Instance types become a "type: pods_per_node" map, with the shared values
in defaults.buildkit so clusters.yaml is the only source:
One Deployment means one pod spec, so those counts are constraints, not
per-node settings: the pod takes the smallest size any entry allows,
making "every type holds at least its declared count" true by
construction. Placement is then bin-packing.
Both arches change shape, not just amd64: the per-size instance-type
label and taint are dropped from each NodePool because they cannot
describe a multi-size pool. workload/buildkit- already isolates
them, nodeSelector now pins arch via kubernetes.io/arch, and the pod
tolerates instance-type with Exists so it still schedules onto existing
nodes that carry the old taint. arm64 keeps a single instance type today
but picks up the same label/taint/selector change.
Unchanged for both arches: pod resources (42c/155Gi amd64, 14c/51Gi
arm64), NodePool limits, and autoscaling.yaml is byte-identical. KEDA is
unaffected — min/max are replica counts and the pod spec is untouched.