Skip to content

Invalidate CloudFront at the end of a blue-green deploy - #2254

Closed
partouf wants to merge 1 commit into
mainfrom
blue-green-cloudfront-invalidation
Closed

partouf wants to merge 1 commit into
mainfrom
blue-green-cloudfront-invalidation

Conversation

@partouf

@partouf partouf commented Jul 29, 2026

Copy link
Copy Markdown
Member

Blue-green deploys switched traffic and updated the compiler routing table but never touched CloudFront, so the edge kept serving the previous version's assets until they expired. environment refresh was the only path that invalidated.

Invalidation now runs as step 6.5, after the compiler routing update and before the GitHub notifications, using the same per-environment config as environment refresh: prod invalidates /* on godbolt.org, compiler-explorer.com and godbo.lt; other environments invalidate their path prefix; aarch64 and wintest have no entries so it's a no-op there. Failures are already swallowed per-distribution inside invalidate_cloudfront_distributions, so this can't abort a deploy that has already switched traffic.

Skip with --skip-cloudfront, mirroring the flag on environment refresh.

blue-green switch and blue-green rollback still don't invalidate; rollback in particular may want the same treatment.

🤖 Generated with Claude Code

Blue-green deploys switched traffic and updated compiler routing but left
the edge caches alone, so godbolt.org kept serving the previous version's
assets until they expired. Only `environment refresh` invalidated.

Runs after the compiler routing update, using the same per-environment
config as `environment refresh`: prod invalidates /* across all three
domains, other environments just their path prefix. Skip with
--skip-cloudfront.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mattgodbolt

mattgodbolt commented Aug 5, 2026

Copy link
Copy Markdown
Member

kept serving the previous version's assets until they expired.

Which assets specifically? Many of the assets have uniquifiers on them so invalidating everything might be overkill. I was hoping we could avoid the invalidtion but obviously we used to do it, so I'm not opposed but wonder if we can target it at just the root path or something?

@partouf

partouf commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

kept serving the previous version's assets until they expired.

Which assets specifically? Many of the assets have uniquifiers on them so invalidating everything might be overkill. I was hoping we could avoid the invalidtion but obviously we used to do it, so I'm not opposed but wonder if we can target it at just the root path or something?

I think in most cases it just works fine and it's mostly a precaution now that I've been doing every release when there are new compilers and libraries. Of course since they're now lazy loaded that shouldn't be needed either anymore. So maybe this PR isn't needed either and I don't really have to invalidate at all anymore.

Will have to do a big release at some point and test without invalidation.

@partouf
partouf marked this pull request as draft August 5, 2026 12:04
@partouf

partouf commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

kept serving the previous version's assets until they expired.

this is ofcourse bot-talk and it doesn't understand what it's talking about.

@partouf

partouf commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

The issue lied mostly with client-options.js which when GETted would just serve you whatever instance you hit through the LB whether or not the query param with the hash would actually match the hash of the CE build.
And that in turn would trigger cloudfront to cache an old version of the compiler list under a new hash for example.

Was that already over since we switched to blue-green? - I'm not sure. The turnover isn't instant and it works a little confusing.

BUT - I think that these kinds of static-y API calls have a 10 minute? cache, so that would leave 10 minutes in a mixed state. So if you refresh after the blue-green deploy or even after the physical switchover, it will still be half old half new.

I think for normal situations its perfectly fine to have a cache on the api endpoints until you actually deploy a new version.
Or you need to wait 10 minutes before doing the GH notices and hope for the best nothing clashes. (like compilers that don't exist anymore being shown and giving errors etc, or features that don't have backwards/forwards compatibility)

Maybe there's a better way of cruching the numbers on cache and cache paths

partouf added a commit that referenced this pull request Aug 5, 2026
The app no longer serves /client-options.js, so the per-distribution
ordered_cache_behavior for it was matching nothing. Dropped from all
three distributions (godbolt.org, compiler-explorer.com, godbo.lt).

Requests to that path now fall through to the default behavior on the
same ALB origin. The hardcoded cache policy ce5806f2 and origin request
policy 216adef6 were only used here; they aren't terraform-managed so
they remain in AWS unreferenced.

Related to #2254.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@partouf partouf closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants