Created on 09-04-201911:47 AM - edited 09-04-201901:53 PM
My Kubernetes series (Part1, Part2) was strictly focused on MiNiFi on K8S. NiFi and MiNiFi may communicate over Site 2 Site; however, often the pattern is to leverage Kafka for a clean message handoff. NiFi within these patterns is generally the central router and transformer of messages. Think of it like "FedEx" for data. Till now most have deployed NiFi on bare metal or VMs. Natural evolution kicks in. Deploy NiFi on k8s and yes it's super simple. In this article I will demonstrate how to deploy both a NiFi and ZooKeeper cluster (none being a single pod!) on Azure Kubernetes Service (AKS); however, all artifacts may be leveraged to deploy on virtually any kubernetes offering.
Some knowledge of Kubernetes and NiFi
AKS / k8s Cluster
NiFi image available in a registry (ie dockerhub)
All the contents in the ymls below can be placed into single file for deployment. For this demonstration, chucking it into smaller components makes it easier to explain.
I have loaded a NiFi image into Azure Container Repository. You can use the NiFi image available here in DockerHub.
NiFi uses ZooKeeper for several state management functions. More on that here. ZooKeeper for NiFi can be deployed using embedded or stand alone mode. Here I will deploy 3 pods of ZK on k8s. Deploying ZK on k8s is super simple.