About haris_khan

haris_khan · ‎04-13-2020

Thanks Sunil. I did happen to try this the other day. I did it through the helm charts - cetic/nifi and faced this scenario. I started with 2 pod cluster and killed one of the pods and Nifi cluster became read-only but as soon as I brought up another pod, the cluster became normal. Probably this happened because replicas are managed through statefulsets, so when the pod came back it has retained the same properties and volume mounts etc. So I think scaling cluster down might be an issue considering this property of Nifi. Surely I will start looking into Minifi, it has always been of interest but never looked in detail... Thanks 🙂

haris_khan · ‎04-11-2020

Hello.. This is a very informative article. I just have a question in terms of creating a Nifi cluster on k8s.. In case of auto-scaling or k8s launching a new pod, if a pod/node crashes, how does Nifi maintains consistency. We currently use Nifi in cluster mode on VMs but when any node goes down, we have to manually disconnect the failed node from cluster or have to bring up the failed node to get the cluster in edit-able mode i.e. dataflows can be edited from UI. As I understand, Nifi does this to maintain consistency across all nodes. So my question is in K8s kind of environment, where pods are created and destroyed very quickly.. how will Nifi cluster behave ? Sorry for writing in a old article, will not mind to start a new thread but just thought the question is relevant here. Sorry I have limited understanding of Kubernetes... Thanks 🙂

haris_khan · ‎05-25-2019

Hello Experts, I wanted to know if there is a limit on number of processors that can be using in a Nifi Cluster. To explain further, I have a Nifi Dataflow which can only consume for a Single Kafka Topic due to transactional consistency requirements. Now to handle consumption from multiple Kafka Topics, I have to replicate same flow (basically re-import same template) multiple times. Due to this I could potentially need 20k processors on a single Nifi cluster. Although they are grouped properly in separate process groups. Is there a threshold on number of processors that can be used ? Or can we derive such limit based on input data volume or the kind of processors I am using. PS: We are trying to work on using same Nifi Dataflow to cosume from multiple Topics but due to transactional sequencing and completeness requirements, it is getting over complex and might take some more time to develop. For now we have to go with replicating same flow many times. Thanks

Online	Offline
Last Visited	‎04-13-2020 12:50 PM

Member Since	‎05-25-2019 12:06 PM
Last Visited	‎04-13-2020 12:50 PM
Posts	4

Cloudera Community

Re: Deploy NiFi On Kubernetes

Re: Deploy NiFi On Kubernetes

Limit of maximum number of processors in Nifi