Currently you just copy the NAR to the lib directory of every server and restart NiFi, you could script this process. The NiFi community has talked about a future enhancement to have an extension registry where custom NARs could be uploaded, and then a given NiFi instance/cluster could load NARs from the registry. There is still a lot of design to be done, but this could potentially help with this scenario. ... View more

Do you still have the original source of data outside of NiFi? If so, you could completely wipe your NiFi back to a clean slate by stoping NiFi and deleting all of the "_repository" directories and restarting. Then you could configure appropriate back-pressure before trying to pick up these files again. If that is not an option you probably need to figure out what is causing things to hang, is it going out of memory and showing exceptions in the nifi-app.log? ... View more

There is no hard-coded limit, but there is definitely some limit in terms of performance. Flow File attributes are held in memory (in addition to being persisted to disk) so more attributes means more Java objects on the heap, means more garbage collection pressure. I don't think there is any way to set what the limit is because it depends on how much data is in the attributes, how much memory you have, how many flow files, etc. ... View more

Here is what you would need to do to configure PublishKafka to talk to a kerberized Kafka... 1) You can either rely on /etc/krb5.conf or you can tell NiFi to use a specific krb5.conf by setting nifi.kerberos.krb5.file= in nifi.properties to point to some other krb5.conf file. 2) Create a JAAS file lets say kafka-jaas.conf with the following: KafkaClient {   com.sun.security.auth.module.Krb5LoginModule required   useKeyTab=true   storeKey=true   keyTab="/path/to/nifi-iotdemot.keytab"   serviceName="kafka"   principal="nifi/iotdemo.field.hortonworks.com@LAKE"; }; Changing the keyTab path to the appropriate path. 3) Specify this in NiFi's bootstrap.conf: java.arg.15=-Djava.security.auth.login.config=/path/to/kafka-jaas.conf 4) Configure PublishKafka: Security Protocol = PLAINTEXTSASL Service Name = kafka The service name should match what is in the JAAS file above. You don't need to do any of the other stuff related to ZooKeeper, unless your NiFi instance is also using a Kerberized ZooKeeper for NiFi's state management. The above steps are what you need to do only for NiFi talking to Kafka. ... View more

That means the user you are logging in as does not have permission to access the UI. You can check nifi-user.log to see the user identity that is coming from your request (it should be the DN of your cert) and compare that to what is in users.xml and authorizations.xml. If this is your "initial admin" identity then this should have been entered in authorizers.xml as the initial admin, and that would have granted it all the correct permissions. If you had already tried to setup an initial admin before then you need to delete users.xml and authorizations.xml before trying to change the "initial admin", otherwise it won't take effect. ... View more