How to rename or copy and rename nested fields?

[Ants5]

`

[INPUT]
    Name                tail
    Tag                 charging-us-prod-vmlogs.*
    Path                /var/log/containers/hes-hes-2030*_charging-us-prod_*.log
    Parser              cri
    DB                  /var/fluent-bit/state/charging-us-prod-vmlogs.db
    Mem_Buf_Limit       100MB
    Skip_Long_Lines     On
    Skip_Empty_Lines    On
    Refresh_Interval    5
    Rotate_Wait         20
    storage.type        filesystem
    #Read_from_Head      off

[FILTER]
    Name                kubernetes
    Match               charging-us-prod-vmlogs.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     charging-us-prod-vmlogs.var.log.containers.
    Merge_Log           On
    Keep_Log            On
    namespace_labels    On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off
[FILTER]
    Name                nest
    Match               charging-us-prod-vmlogs.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          kubernetes.
[FILTER]
    Name                record_modifier
    Match               charging-us-prod-vmlogs.*
    Record              area   usa
    Record              type   test
    Remove_key          kubernetes.pod_id
    Remove_key          kubernetes.pod_ip
    Remove_key          kubernetes.docker_id
    Remove_key          kubernetes.container_hash
    Remove_key          kubernetes.container_image
    #Remove_key          kubernetes.container_name
    #Remove_key          kubernetes.pod_name
    #Remove_key          kubernetes.namespace_name
[FILTER]
    Name                modify
    Match               charging-us-prod-vmlogs.*
    Rename              kubernetes.container_name   service_name
    Rename              kubernetes.namespace_name   namespace
    Rename              kubernetes.pod_name   pod
    Rename              kubernetes.host   nodename
    #Set                 cluster ${kubernetes_namespace.labels.cluster}
    #Set                 cluster ${kubernetes_namespace.labels.project}
    #Rename              kubernetes_namespace.labels.cluster  cluster
    #Rename              kubernetes_namespace.labels.project  project
    #Remove              kubernetes_namespace

#[FILTER]
#    Name                nest
#    Match               charging-us-prod-vmlogs.*
#    Operation           nest
#    Wildcard            kubernetes.*
#    Nest_under          kubernetes
#    Remove_prefix       kubernetes.

[OUTPUT]
    Name                stdout
    Match               charging-us-prod-vmlogs.*
[OUTPUT]
    Name                http
    Match               charging-us-prod-vmlogs.*
    host                localhost
    port                8427
    http_user           aaaaa
    http_passwd         123123
    header              AccountID 0
    header              ProjectID 0
    uri                 /insert/jsonline?_stream_fields=project,cluster,namespace,service_name,type&_msg_field=log
    format              json_lines
    json_date_format    iso8601
    compress            gzip

`

My data collection configuration is shown above. The Kubernetes plugin has namespace_labels enabled to retrieve labels from the namespace, but the retrieved labels are in nested JSON format. How can I rename kubernetes_namespace.labels.cluster to the cluster name without unnesting the tags? My log format is as follows.

charging-us-prod-vmlogs.var.log.containers.hes-hes-2030-795697479c-ps9dc_charging-us-prod_hes-hes-2030-102360dc7fd5eaab6fe5b984d1a742884ed36bb43864328b284e174a3d8da43c.log: [[1767168962.880809955, {}], {"stream"=>"stdout", "logtag"=>"F", "log"=>"{"@timestamp":"2025-12-31T08:16:02.880Z","caller":"util/httputil.go:130","content":"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:\"[Subscription xmlns=\\\"urn:ieee:std:2030.5:ns\\\"]\\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\\n [encoding]0[/encoding]\\n [level]+S1[/level]\\n [limit]0[/limit]\\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_2030_svc/ntfy/synergy/2F266B0212F96DF43FCD4AF2E51B007D00060981[/notificationURI]\\n[/Subscription]\", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165","level":"info"}", "@timestamp"=>"2025-12-31T08:16:02.880Z", "caller"=>"util/httputil.go:130", "content"=>"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:"[Subscription xmlns=\"urn:ieee:std:2030.5:ns\"]\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\n [encoding]0[/encoding]\n [level]+S1[/level]\n [limit]0[/limit]\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_20_svc/ntfy/synergy/2F266B0212F43FCD4AF2EB007D00060981[/notificationURI]\n[/Subscription]", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165", "level"=>"info", "kubernetes_namespace"=>{"name"=>"charging-us-prod", "labels"=>{"cluster"=>"aiot-us-prod", "dynakube.internal.dynatrace.com/instance"=>"aiot-qa", "field.cattle.io/projectId"=>"p-7w2zz", "kubernetes.io/metadata.name"=>"charging-us-prod", "project"=>"charging"}}, "pod"=>"hea-hes-22230-795697479c-ps9dc", "namespace"=>"charging-us-prod", "nodename"=>"ip-11-11-11-153.us-east-2.compute.internal", "service_name"=>"hea-hes-22230", "area"=>"usa", "type"=>"test"}]

[patrick-stephens]

Personally I'd probably just do it all in one go with a LUA filter, or even better a processor. You can write some nice LUA that will do everything you need in one pass plus cope with defaults, missing information, etc. as required.

Chaining lots of filters together does have an impact: each filter needs to unpack then re-pack the data, it runs on a single thread.

If you switch to YAML format you can write a multiline LUA processor all in-line in the config: https://docs.fluentbit.io/manual/data-pipeline/processors/filters#lua

If you are using the helm chart, it is fairly trivial to provide LUA filters as well with the classic config: https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit#using-lua-scripts