What version of NiFi and what version of Kafka? NIFi 1.x has GetKafka for Kafka 0.8, ConsumeKafka for Kafka 0.9, and ConsumeKafka_0_10 for Kafka 0.10. Whenever possible the matching processor should be used with the matching broker. I believe all of them have some kind of property that controls the initial offset for the first time the processor is ever started, basically saying whether to start at the beginning of the topic, or at the latest offset. After that it is always going to use the last offset that the Kafka client has consumed in order to never miss data. If you ever want to start back over at the current time I believe you can just change to a new consumer group id with Offset Reset set to latest. ... View more

You need to configure NiFi to use a login identity provider such as LDAP or Kerberos, and then navigate to the secure URL of the NiFi web ui, for example https://localhost:8443/nifi. If you have a certificate in your browser and get prompted, make sure to decline otherwise it will use that to authenticate. There are many reasons covering how to configure LDAP or Kerberos: https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/ http://bryanbende.com/development/2016/08/31/apache-nifi-1.0.0-kerberos-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html ... View more

You should use a MergeContent processor before PutHDFS to merge flow files together based on a minimum size. ... View more

It is hard to tell without being able to see your code, but it seems like this is saying that in LookupProcessor you have a PropertyDescriptor that represents a Controller Service, but the class you specified is not the interface from nifi-customservice-api-nar, but instead it is the implementation from nifi-customservice-nar. As an example, InvokeHttp uses SSLContextService: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L205 SSLContextService is the interface that comes from nifi-standard-services-api: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-service-api/src/main/java/org/apache/nifi/ssl/SSLContextService.java The implementation is StandardSSLContextService which is is nifi-ssl-context-bundle: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java ... View more

What protocol is the data producer using to push data to the first NiFi instance? ... View more