Fluent Bit in IBM Kubernetes Service with Humio

Ong Khai Wei
3 min readJul 31, 2020

Log is essential for developer and system engineer to understand the behaviour of application before any troubleshooting can be done. When application behaves normally, it kinda useless and occupying valuable resources such as memory and disk space. When user encountered error in using application, log becomes main source of information to assess and triage for troubleshooting. Recently I was asked to explore Humio log management solution.

Humio leverages many Open Source tools for the integration. Fluent-bit is recommended as the log processor and forwarder to pipe the log to Humio, today I am going to cover similar topic as per my article about LogDNA configuration on IBM Cloud.

Similarly to the previous article, I have followed the instruction to deploy on Fluent-Bit, but there is no log comes into Humio server. Then I realized the path of Kubernetes log is different on IBM Kubernetes Service, therefore have to do some modification on DaemonSet of Fluent-Bit. As I deployed Fluent-Bit via Helm Chart, therefore I will need to modify the yaml directly in Kubernetes cluster

spec:
volumes:
...
- name: vardatakubeletlogs
hostPath:
path: /var/data/kubeletlogd
...
containers:
- name: humio
image: 'fluent/fluent-bit:1.4.2'
...
volumeMounts:
- name: vardatakubeletlogs
mountPath: /var/data/kubeletlogs
...

Humio provides a Elastic Search interface to accept the log. However when Humio is deployed on OpenShift Cluster, the route only expose by either port 80 or 443 (Elastic Search uses port 9200). This is where you can create a secure route from OpenShift and override the value in Helm Chart.

Name: <ROUTE_OF_HUMIO_CORE_ES>
Service: humio-humio-core-es
Target Port: 9200 -> es (TCP)
TLS Termination: Edge
Insecure Traffic: None

The rest of the value can leave it as empty and press Create button.

New route will be created and we can replace the host value in yaml file.

In humio-agent.yaml file, then we can replaced with the correct value as below:

---
humio-core:
enabled: false
humio-fluentbit:
enabled: true
humioRepoName: default // NAME_OF_API_TOKEN
humioHostname: <ROUTE_OF_HUMIO_CORE_ES>
tokenSecretName: fluentbit-shared-token
es:
port: 443
tls: true
tls_verify: false

And the command to deploy Fluent-bit will be below:

helm install humio humio/humio-helm-charts --namespace logging -f humio-agent.yaml

--

--