Running Kafka in Kubernetes

Deploying and running successfully Kafka in Kubernetes is not the purpose of the guide, however it is interesting to share some tips about this task which can be quite complex.

Confluent platform with helm

Confluent provides a very interesting set of configurations with helm that you can use to setup and build your infrastructure in Kubernetes:

• https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html
• https://github.com/confluentinc/cp-helm-charts

The templates provides in this guide are built on top of the Confluent platform and these configurations, which however can be adapted to run on any kind of deployment.

Testing with minikube

Make sure you start minikube with enough memory and cpu resources, example:

minikube start --memory 8096 --cpus 4

Before starting the deployment with helm, you can use the following configuration to create the require storage classes:

minikube_storageclasses.yml

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kafka-broker
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kafka-zookeeper
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain

Then apply:

kubectl create -f minikube_storageclasses.yml

Modify the values.yml to include the storage classes and some restrictions on the containers to get it successful:

values.yaml

## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
cp-zookeeper:
  enabled: true
  servers: 3
  image: confluentinc/cp-zookeeper
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  persistence:
    enabled: true
    ## The size of the PersistentVolume to allocate to each Zookeeper Pod in the StatefulSet. For
    ## production servers this number should likely be much larger.
    ##
    ## Size for Data dir, where ZooKeeper will store the in-memory database snapshots.
    dataDirSize: 5Gi
    dataDirStorageClass: "kafka-zookeeper"

    ## Size for data log dir, which is a dedicated log device to be used, and helps avoid competition between logging and snaphots.
    dataLogDirSize: 5Gi
    dataLogDirStorageClass: "kafka-zookeeper"
  resources:
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
    limits:
     cpu: 100m
     memory: 256Mi
    requests:
     cpu: 100m
     memory: 256Mi

## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
cp-kafka:
  enabled: true
  brokers: 3
  image: confluentinc/cp-kafka
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  persistence:
    enabled: true
    storageClass: "kafka-broker"
    size: 5Gi
    disksPerBroker: 1
  resources:
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
    limits:
     cpu: 200m
     memory: 512Mi
    requests:
     cpu: 200m
     memory: 512Mi

## ------------------------------------------------------
## Schema Registry
## ------------------------------------------------------
cp-schema-registry:
  enabled: true
  image: confluentinc/cp-schema-registry
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  resources:
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
    limits:
     cpu: 100m
     memory: 512Mi
    requests:
     cpu: 100m
     memory: 512Mi

## ------------------------------------------------------
## REST Proxy
## ------------------------------------------------------
cp-kafka-rest:
  enabled: true
  image: confluentinc/cp-kafka-rest
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  resources:
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
    limits:
     cpu: 100m
     memory: 256Mi
    requests:
     cpu: 100m
     memory: 256Mi

## ------------------------------------------------------
## Kafka Connect
## ------------------------------------------------------
cp-kafka-connect:
  enabled: true
  image: confluentinc/cp-kafka-connect
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
    #limits:
     #cpu: 100m
     #memory: 512Mi
    #requests:
     #cpu: 100m
     #memory: 512Mi

## ------------------------------------------------------
## KSQL Server
## ------------------------------------------------------
cp-ksql-server:
  enabled: true
  image: confluentinc/cp-ksql-server
  imageTag: 5.0.1
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  ksql:
    headless: false

Starting helm:

The templates provided are built on the naming convention of a helm installion called “confluent-oss” in a name space called “kafka”:

helm3 command:

helm install confluent-oss confluentinc/cp-helm-charts

The following command is valid for helm2 only, and left for historical purposes

helm install cp-helm-charts --name confluent-oss --namespace kafka

The helm installation provided by Confluent will create:

• Zookeeper cluster in a statefulSet
• Kafka Brokers cluster in a statefulSet
• Kafka Connect in a Deployment
• Confluent Schema registry in a Deployment
• Confluent ksql-server in a Deployment
• Confluent kafka-rest in a Deployment