Humio on Redhat OpenShift Container Platform

6 min readFeb 17, 2021

A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user. It becomes more and more widely adopted by community. In this case, Humio also changed its own of deployment from Helm chart to Operator — https://www.humio.com/whats-new/blog/how-to-use-the-humio-operator-to-run-humio-on-kubernetes

While the blog post in Humio website illustrates the step for any Kubernetes platform, for this post, deployment on RedHat OpenShift Container Platform (OCP) will be the main target. In general, the steps are similar, however there are a few steps that are different.

Pre-requisite:

RedHat OpenShift Cluster
OC CLI installed in your laptop
Helm v3 CLI installed in your laptop

I will skip the steps of setting up RedHat OpenShift Container Platform, OC CLI and Helm. There are multiple ways to have RedHat OpenShift cluster, in this example I am using Red Hat OpenShift on IBM Cloud. OC CLI and Helm CLI are available from Internet easily.

Summary of steps

Install CRDs
Create Project / Namespace
Install the Humio Operator
Configure Service Account Privileges for Humio Operator
Install Strimzi Kafka
Install Certificate Manager
Create a Humio Cluster
Configure Service Account Privileges for Humio Cluster
Create Route for Console and ElasticSearch endpoint

Step 1 — Install CRDs

Find the version of the Humio Operator that we want to install, and create the CRDs. In this case I choose the latest version which is 0.5.0

export HUMIO_OPERATOR_VERSION=0.5.0
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humioclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humioexternalclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humioingesttokens.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humioparsers.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humiorepositories.yaml
kubectl apply -f https://raw.githubusercontent.com/humio/humio-operator/operator-$HUMIO_OPERATOR_VERSION/config/crd/bases/core.humio.com_humioviews.yaml

Step 2 — Create Project / Namespace

This is create namespace or project that will host all the components related to Humio. Choose a suitable name, in this case I choose logging .

oc new-project logging

Step 3 — Install the Humio Operator

Humio Operator is installed via Helm command. Since we have exported environment variable of HUMIO_OPERATOR_VERSION in the previous step, then we just re-use the same variable.

helm install humio-operator humio-operator/humio-operator \
 — namespace logging \
 — version=”$HUMIO_OPERATOR_VERSION”

Step 4 — Configure Service Account Privileges for Humio Operator

By default, RedHat OpenShift Container Platform hare more tighten security control for service account. This may result pod failed to be schedule due to insufficient privilege. To check the status of Humio Operator pod, you can check via oc get pods -n logging , if the pod is not available, it may due to privilege of service account humio-operator. You can grant higher privilege access to service account.

oc adm policy add-scc-to-user privileged system:serviceaccount:logging:humio-operator -n logging

Humio Operator in RedHat OpenShift Container Platform Console

Step 5—Install Strimzi Kafka

Humio requires access to Kafka instance in order to buffer the incoming log that streaming in before processing. User can either re-use existing Kafka instance, or using Kafka-as-a-Service from Cloud Provider such as AWS. In this example I showcase how to install Strimzi Kafka from Operator Hub.

Navigate to OperatorHub in RedHat OpenShift Container Platform and search for strimzi

Click install button to proceed with the installation

Installation mode change to A specific namespace on the cluster and choose logging . Click Install button to proceed to install Operator.

After a few minutes, Strimzi Operator will be installed successfully.

Click on the instance of Operator, navigate to Kafka tab. At this moment it will be empty. Click Create Kafka button to proceed to provision a Kafka instance.

Give a meaningful name of your Kafka instance. I gave a name of humio-cluster . Click Install button to proceed.

It will take a few minutes to provision a 3 nodes Kafka cluster, together with Zookeeper. This makes the setup of Kafka with HA easy.

Once Kafka is fully up and running, we will need to extract some information to be used later for Humio cluster setup.

oc get services -n logging

We need 2 information from service — service name for Kafka bootstrap humio-cluster-kafka-bootstrapand its port 9092, and service name for Zookeeper humio-cluster-zookeeper-client and its port 2181.

Step 6 — Install Certificate Manager

This can be done easily with the following command:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v0.16.0 \
  --set installCRDs=true

Please take note that Humio cannot be installed along with IBM Common Services due to conflict with cert-webhook.

Step 7 — Create a Humio Cluster

Now Humio cluster can be deployed via CRD. HumioCluster CRD will automatically start up Humio pods. We need to use the extracted information from our existing ZooKeeper and Kafka clusters into yaml file as per follow:

apiVersion: core.humio.com/v1alpha1
kind: HumioCluster
metadata:
name: humiocluster
namespace: logging
spec:
  autoRebalancePartitions: true
  tls:
    enabled: false
  dataVolumePersistentVolumeClaimSpecTemplate:
    storageClassName: ibmc-block-gold
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 10Gi
  environmentVariables:
    - name: "SINGLE_USER_PASSWORD"
      value: "develop3r"
    - name: "HUMIO_JVM_ARGS"
      value: "-Xss2m -Xms256m -Xmx1536m -server -XX:+UseParallelOldGC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Dzookeeper.client.secure=false"
    - name: "ZOOKEEPER_URL"
      value: "humio-cluster-zookeeper-client.logging.svc:9092"
    - name: "KAFKA_SERVERS"
      value: "humio-cluster-kafka-bootstrap.logging.svc:9092"

Step 8— Configure Service Account Privileges for Humio Cluster

Once again you may encounter situation where Humio pods is not start-up. Then you can grant higher privileges for services account humio-cluster :

oc adm policy add-scc-to-user privileged system:serviceaccount:logging:humiocluster-humio -n logging

Step 9 — Create Route for Console and ElasticSearch endpoint

By now, all Humio components are up and running. The last step will be exposed the service so that we can access the web console and ElasticSearch interface to accept log streaming. Replace <FQDM_OF_THIS_ROTE>with the domain of your RedHat OpenShift Container Platform.

To create route for console — humioconsole-route.yaml :

apiVersion: v1
kind: Route
metadata:
  name: humio-console
  namespace: logging
spec:
  host: <FQDM_OF_THIS_ROUTE>
  to:
    kind: Service
    name: humiocluster
  port:
    targetPort: http
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow

To create route for ElasticSearch endpoint— humiocluster-es-route.yaml :

apiVersion: v1
kind: Route
metadata:
  name: humio-es
  namespace: logging
spec:
  host: <FQDM_OF_THIS_ROUTE>
  to:
    kind: Service
    name: humiocluster
  port:
    targetPort: es
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow