Get logs only from a specific namespace in fluent-bit

[shammi-stl]

Hi all, I'm trying to retrieve logs from pods running in a specific namespace, from a cluster that has multiple namespaces. Im currently able to receive logs from all these namespaces to an index called 'logstash', my goal is to create an index called 'staging-logstash' to get logs from my namespace 'stl-staging' only and I want other namespaces to send their logs to logstash index as usual.

I have tried the following config and performed restarts on the fluentbit daemonset, the 'staging-logstash' index does not appear to be creating in elasticsearch and I'm having trouble identifying why.

apiVersion: v1
data:
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        Annotations         Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

    [FILTER]
        Name                rewrite_tag
        Match               kube.*
        Rule                $kubernetes['namespace_name'] ^stl-staging$ staging.kube false
        Emitter_Name        re_emitted_staging
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            ${HOST}
        Port            9200
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Suppress_Type_Name  On
        tls             On
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
    [OUTPUT]
        Name            es
        Match           staging.*
        Host            ${HOST}
        Port            9200
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Suppress_Type_Name  On
        tls             On
        Logstash_Format On
        Logstash_Prefix staging-logstash
        Replace_Dots    On
        Retry_Limit     False
  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
kind: ConfigMap

I get the following output for this API call GET _cat/indices/staging-logstash-*?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size

My fluentbit version is fluent/fluent-bit:1.8 I would greatly appreciate if someone could point out the problem im having.

Thanks.

[yudin-s]

The easiest way to make this reliable is to split the namespace at the Tail input, because Kubernetes container log filenames already contain the namespace.

Your current rewrite_tag approach only works if the Kubernetes filter has already added kubernetes.namespace_name and the rule is actually matching. Since the staging-logstash-* index is not created at all, first assume the rule is not producing any staging.* records.

A simpler shape is two Tail inputs with different tags and different DB files:

[INPUT]
    Name              tail
    Tag               staging.*
    Path              /var/log/containers/*_stl-staging_*.log
    Parser            docker
    DB                /var/log/flb_staging.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Exclude_Path      /var/log/containers/*_stl-staging_*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

Then keep your outputs separated by tag:

[OUTPUT]
    Name            es
    Match           staging.*
    Logstash_Format On
    Logstash_Prefix staging-logstash
    ...

[OUTPUT]
    Name            es
    Match           kube.*
    Logstash_Format On
    ...

That avoids depending on Kubernetes metadata just to decide the index.

If you want to keep rewrite_tag, add a temporary stdout output to prove the rule is firing:

[OUTPUT]
    Name  stdout
    Match staging.*

and run with Log_Level debug. If stdout shows nothing for staging pods, the issue is before Elasticsearch: either the Kubernetes metadata field is not present, or the record accessor in the Rule is not matching on Fluent Bit 1.8.

Also note that with Logstash_Format On, the actual index name will be date-suffixed, like staging-logstash-YYYY.MM.DD, not exactly staging-logstash.

If this solves it, please mark this comment as the answer so others can find it faster.

[patrick-stephens]

Yeah that's how I'd do it as it also means you have separate inputs for configuration and dealing with noisy neighbours, etc.

One thing to note as well is you will need to set the Kube_Tag_Prefix correctly if you want to use the kubernetes filter with the staging.* records: the prefix is effectively stripped off but assumes a default based on kube.*.

https://www.telemetryforge.io/blog/kubernetes-logging