We are currently using HDF (Hortonworks Dataflow) 3.3.1 which bundles Kafka 2.0.0. Problem is with running multiple connectors with different configuration(Kerberos principals) on same KafkaConnect Cluster.
As part of this Kafka version, all connectors are supposed to use same consumer/producer properties which have been set in worker configuration with consumer.* or producer.* prefix. But as I stated, we have multiple users (apps) running their own connectors and we can't use a single Kerberos principal to allow read on all topics.
So just wanted to check with experts if there is any way this security limitation can be over come. The option I can think of is - run a different Kafka-Connect cluster for each Kafka User (different principals) but what implications it could have if we run many KafkaConnect Clusters on same nodes ? Will it cause any impacts in term of resources (Java heap etc.) or this is the only way (standard procedure) to handle this.
PS: In later releases (2.3+) this problem is fixed via KAFKA-8265 and these settings can be overwritten but even if we try upgrading to latest HDF we will only get Kafka 2.1 which will not solve this issue.
Thanks for your help !!
... View more
Hello Experts, I wanted to know if there is a limit on number of processors that can be using in a Nifi Cluster. To explain further, I have a Nifi Dataflow which can only consume for a Single Kafka Topic due to transactional consistency requirements. Now to handle consumption from multiple Kafka Topics, I have to replicate same flow (basically re-import same template) multiple times. Due to this I could potentially need 20k processors on a single Nifi cluster. Although they are grouped properly in separate process groups. Is there a threshold on number of processors that can be used ? Or can we derive such limit based on input data volume or the kind of processors I am using. PS: We are trying to work on using same Nifi Dataflow to cosume from multiple Topics but due to transactional sequencing and completeness requirements, it is getting over complex and might take some more time to develop. For now we have to go with replicating same flow many times. Thanks
... View more