Skip to content

How to rename or copy and rename nested fields? #11324

Description

@Ants5

`

[INPUT]
    Name                tail
    Tag                 charging-us-prod-vmlogs.*
    Path                /var/log/containers/hes-hes-2030*_charging-us-prod_*.log
    Parser              cri
    DB                  /var/fluent-bit/state/charging-us-prod-vmlogs.db
    Mem_Buf_Limit       100MB
    Skip_Long_Lines     On
    Skip_Empty_Lines    On
    Refresh_Interval    5
    Rotate_Wait         20
    storage.type        filesystem
    #Read_from_Head      off

[FILTER]
    Name                kubernetes
    Match               charging-us-prod-vmlogs.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     charging-us-prod-vmlogs.var.log.containers.
    Merge_Log           On
    Keep_Log            On
    namespace_labels    On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off
[FILTER]
    Name                nest
    Match               charging-us-prod-vmlogs.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          kubernetes.
[FILTER]
    Name                record_modifier
    Match               charging-us-prod-vmlogs.*
    Record              area   usa
    Record              type   test
    Remove_key          kubernetes.pod_id
    Remove_key          kubernetes.pod_ip
    Remove_key          kubernetes.docker_id
    Remove_key          kubernetes.container_hash
    Remove_key          kubernetes.container_image
    #Remove_key          kubernetes.container_name
    #Remove_key          kubernetes.pod_name
    #Remove_key          kubernetes.namespace_name
[FILTER]
    Name                modify
    Match               charging-us-prod-vmlogs.*
    Rename              kubernetes.container_name   service_name
    Rename              kubernetes.namespace_name   namespace
    Rename              kubernetes.pod_name   pod
    Rename              kubernetes.host   nodename
    #Set                 cluster ${kubernetes_namespace.labels.cluster}
    #Set                 cluster ${kubernetes_namespace.labels.project}
    #Rename              kubernetes_namespace.labels.cluster  cluster
    #Rename              kubernetes_namespace.labels.project  project
    #Remove              kubernetes_namespace

#[FILTER]
#    Name                nest
#    Match               charging-us-prod-vmlogs.*
#    Operation           nest
#    Wildcard            kubernetes.*
#    Nest_under          kubernetes
#    Remove_prefix       kubernetes.

[OUTPUT]
    Name                stdout
    Match               charging-us-prod-vmlogs.*
[OUTPUT]
    Name                http
    Match               charging-us-prod-vmlogs.*
    host                localhost
    port                8427
    http_user           aaaaa
    http_passwd         123123
    header              AccountID 0
    header              ProjectID 0
    uri                 /insert/jsonline?_stream_fields=project,cluster,namespace,service_name,type&_msg_field=log
    format              json_lines
    json_date_format    iso8601
    compress            gzip

`

My data collection configuration is shown above. The Kubernetes plugin has namespace_labels enabled to retrieve labels from the namespace, but the retrieved labels are in nested JSON format. How can I rename kubernetes_namespace.labels.cluster to the cluster name without unnesting the tags? My log format is as follows.

charging-us-prod-vmlogs.var.log.containers.hes-hes-2030-795697479c-ps9dc_charging-us-prod_hes-hes-2030-102360dc7fd5eaab6fe5b984d1a742884ed36bb43864328b284e174a3d8da43c.log: [[1767168962.880809955, {}], {"stream"=>"stdout", "logtag"=>"F", "log"=>"{"@timestamp":"2025-12-31T08:16:02.880Z","caller":"util/httputil.go:130","content":"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:\"[Subscription xmlns=\\\"urn:ieee:std:2030.5:ns\\\"]\\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\\n [encoding]0[/encoding]\\n [level]+S1[/level]\\n [limit]0[/limit]\\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_2030_svc/ntfy/synergy/2F266B0212F96DF43FCD4AF2E51B007D00060981[/notificationURI]\\n[/Subscription]\", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165","level":"info"}", "@timestamp"=>"2025-12-31T08:16:02.880Z", "caller"=>"util/httputil.go:130", "content"=>"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:"[Subscription xmlns=\"urn:ieee:std:2030.5:ns\"]\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\n [encoding]0[/encoding]\n [level]+S1[/level]\n [limit]0[/limit]\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_20_svc/ntfy/synergy/2F266B0212F43FCD4AF2EB007D00060981[/notificationURI]\n[/Subscription]", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165", "level"=>"info", "kubernetes_namespace"=>{"name"=>"charging-us-prod", "labels"=>{"cluster"=>"aiot-us-prod", "dynakube.internal.dynatrace.com/instance"=>"aiot-qa", "field.cattle.io/projectId"=>"p-7w2zz", "kubernetes.io/metadata.name"=>"charging-us-prod", "project"=>"charging"}}, "pod"=>"hea-hes-22230-795697479c-ps9dc", "namespace"=>"charging-us-prod", "nodename"=>"ip-11-11-11-153.us-east-2.compute.internal", "service_name"=>"hea-hes-22230", "area"=>"usa", "type"=>"test"}]

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions