Fluent Bit in IBM Kubernetes Service with Humio

3 min readJul 31, 2020

Log is essential for developer and system engineer to understand the behaviour of application before any troubleshooting can be done. When application behaves normally, it kinda useless and occupying valuable resources such as memory and disk space. When user encountered error in using application, log becomes main source of information to assess and triage for troubleshooting. Recently I was asked to explore Humio log management solution.

Humio leverages many Open Source tools for the integration. Fluent-bit is recommended as the log processor and forwarder to pipe the log to Humio, today I am going to cover similar topic as per my article about LogDNA configuration on IBM Cloud.

Similarly to the previous article, I have followed the instruction to deploy on Fluent-Bit, but there is no log comes into Humio server. Then I realized the path of Kubernetes log is different on IBM Kubernetes Service, therefore have to do some modification on DaemonSet of Fluent-Bit. As I deployed Fluent-Bit via Helm Chart, therefore I will need to modify the yaml directly in Kubernetes cluster

spec:
  volumes:
    ... 
    - name: vardatakubeletlogs
      hostPath:
       path: /var/data/kubeletlogd
    ...
  containers:
    - name: humio
      image: 'fluent/fluent-bit:1.4.2'
      ...
      volumeMounts:
        - name: vardatakubeletlogs
          mountPath: /var/data/kubeletlogs
      ...

Humio provides a Elastic Search interface to accept the log. However when Humio is deployed on OpenShift Cluster, the route only expose by either port 80 or 443 (Elastic Search uses port 9200). This is where you can create a secure route from OpenShift and override the value in Helm Chart.

Name: <ROUTE_OF_HUMIO_CORE_ES>
Service: humio-humio-core-es
Target Port: 9200 -> es (TCP)
TLS Termination: Edge
Insecure Traffic: None

The rest of the value can leave it as empty and press Create button.

New route will be created and we can replace the host value in yaml file.

In humio-agent.yaml file, then we can replaced with the correct value as below:

---
humio-core:
  enabled: falsehumio-fluentbit:
  enabled: true
  humioRepoName: default // NAME_OF_API_TOKEN
  humioHostname: <ROUTE_OF_HUMIO_CORE_ES>
  tokenSecretName: fluentbit-shared-token
  es:
    port: 443
    tls: true
    tls_verify: false

And the command to deploy Fluent-bit will be below:

helm install humio humio/humio-helm-charts --namespace logging -f humio-agent.yaml

Fluent Bit in IBM Kubernetes Service with Humio

Written by Ong Khai Wei