refactor fastly access logs into shared library, use it for crates.io & docs.rs

[syphar]

in datadog we have some custom inbound processing for access logs, specially fitted to the compute-static access log format.

For docs.rs access logs we would reuse the same format, so we can reuse that logic.

This moves the access log logic into a small shared crate, then used by the compute-static and fastly-compute-docs-rs WASM modules.

This is the first PR, just moving the logic, not changing it.

[syphar]

r? @marcoieni @Turbo87

[syphar]

@Turbo87 super valid comments, this was too quickly done, I'll rework it with your comments in mind

[syphar]

r? @marcoieni @Turbo87

I rebased / redid the PR, as first version just a direct migration of the current behavior, without any change.

Codex review tells me there might be an issue with terragrunt copying the compute-static module, but not having access to the shared library? not sure about the best way to solve it.

[Turbo87]

why did you make this a top-level workspace member?

I'm asking because it seems that the terraform/terragrunt modules currently aren't workspace members for some reason and this seems to balloon the top-level lockfile quite a bit 😅

[syphar]

Only half-reasons you could say.

The CI runs tests / clippy for all workspace crates automatically.
From what I see, we only kept the WASM crates separate because you only can run their tests etc with viceroy.

So my thinking was to add that shared crate to the workspace just because it works without viceroy.

( I don't have a strong opinion here and happily oblige with any other design preference, what do you think?)

[Turbo87]

I see, thanks.

I don't have a strong opinion here and happily oblige with any other design preference, what do you think?

up to the infra team, I guess :D

[ubiratansoares]

I don't have a strong opinion either, I'm fine as it is.

[Turbo87]

Codex review tells me there might be an issue with terragrunt copying the compute-static module, but not having access to the shared library? not sure about the best way to solve it.

hmm, yeah, Claude agrees that this might be problematic. I don't know enough about terragrunt/terraform and the setup here to have an opinion on that part, and I don't think have the credentials to be able to experiment with it either 😅

/cc @marcoieni @ubiratansoares

[ubiratansoares]

@syphar @Turbo87 I ran some plans for both docs-rs (terraform) and crates-io-staging (terragrunt) and I confirmed the issue.

I tried a fix changing the terraform.source for crates.io/staging, ie having instead

terraform {
  source = "${get_repo_root()}//terragrunt/modules/crates-io"
}

and got a successful terragrunt plan

# previous output

20:09:10.641 fastly_service_compute.static: Refreshing state... [id=liljrvY3Xt0CzNk0mpuLa7]
20:09:14.901 fastly_service_dictionary_items.compute_static["compute_static"]: Refreshing state... [id=liljrvY3Xt0CzNk0mpuLa7/0xXJPFIZ3yLoVjrIUTeqY2]
20:09:15.319 Terraform used the selected providers to generate the following execution
20:09:15.319 plan. Resource actions are indicated with the following symbols:
20:09:15.319   ~ update in-place
20:09:15.319 Terraform will perform the following actions:
20:09:15.319   # fastly_service_compute.static will be updated in-place
20:09:15.319   ~ resource "fastly_service_compute" "static" {
20:09:15.319       ~ active_version  = 152 -> (known after apply)
20:09:15.319       ~ cloned_version  = 152 -> (known after apply)
20:09:15.319         id              = "liljrvY3Xt0CzNk0mpuLa7"
20:09:15.319         name            = "static.staging.crates.io"
20:09:15.319         # (6 unchanged attributes hidden)
20:09:15.319       ~ package {
20:09:15.319           ~ source_code_hash = "d3a53d4bea4dc90f1b79f629521972720ed5a221cad1b6e9df18b06a6ab016eeee4c03c8edd9e1867a04bd8569defc765e6e919c80e6fe97121bef9be4214839" -> "b2a245c1d04ae9617135f0bf97c57a8a6abd201ea1d3d0db2b675b3aa12373b83c6b69c1975eec8c60c78037a99478fce327d8fc382c2e4fd3f1747058ff10f9"
20:09:15.319             # (2 unchanged attributes hidden)
20:09:15.319         }
20:09:15.319         # (9 unchanged blocks hidden)
20:09:15.319     }
20:09:15.319 Plan: 0 to add, 1 to change, 0 to destroy.

[syphar]

I tried a fix changing the terraform.source for crates.io/staging, ie having instead

@ubiratansoares I added a commit doing that for staging & prod, is that what you meant?

[ubiratansoares]

I added a commit doing that for staging & prod, is that what you meant?

@syphar Yes. I'll give this PR a review, but I want to merge #1248 and #1249 first

[syphar]

I added a commit doing that for staging & prod, is that what you meant?

@syphar Yes. I'll give this PR a review, but I want to merge #1248 and #1249 first

@ubiratansoares I just rebased this PR, after both PRs above were merged.

@Turbo87 do you want to have another look about the parity in functionality?

For now, this PR tries to have no output / behavior change for crates.io, so we can review improvements / changes separately.

[syphar]

@ubiratansoares on top:
I'm not sure about deploy order between the new access log endpoint & the wasm module, I think @marcoieni didn't do both together last time.

[marcoieni]

I just ran tf apply, nothing fancy. I didn't check if this PR touches more than one module.
If we have a deploy order in the same module, we can have multiple PRs.
By the way, I'm off, so I didn't check the entire conversation, just the last message.

[ubiratansoares]

I just ran tf apply

@marcoieni mind sharing which changes did you deploy?

[marcoieni]

I didn't run anything this week.

I was answering to this:

I think @marcoieni didn't do both together last time.

I don't even know what it refers to "last time". But I don't remember anything in particular. What do you mean with "both" and "last time"?