Hi @Kibrom Gebrehiwot, The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content. To use the attributes of the incoming flow file, you should configure the remote file property with something like: ${path}/${filename} You can find an example here: https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_template.xml Hope this helps. ... View more

Hi @Anshuman Ghosh, It does not answer to all of your questions but you may want to have a look at: http://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka The offset is handled by Kafka itself, in other words when NiFi consumes data from Kafka the offset is committed to Kafka and NiFi does not store it. That's why there is this property "Offset reset" in case there is no offset on Kafka's side. In this case you only have the values proposed by the processor. It corresponds to Kafka's 'auto.offset.reset' property. You may want to look at Kafka's documentation for more details. https://kafka.apache.org/documentation/#newconsumerconfigs Regarding scaling, the link I provided before should give you a good idea. But in short... NiFi does scale very well with Kafka, you can increase the number of threads running in the JVM (Jetty is not involved at all) to consume data from Kafka, but you can also install NiFi in cluster mode to have multiple nodes of NiFi consuming data (and even multiple threads for each one of the nodes of your cluster). Hope this helps. ... View more

Hi @Kibrom Gebrehiwot, You need to set the nifi.kerberos.krb5.file=/etc/krb5.conf property so that NiFi is able to know the information about your realms/controllers and negotiate tickets. Once this property is set, NiFi must be restarted. Then, in the PutHDFS processor, you need to use the principal and keytab of the user you want to use to push data into HDFS. It is recommended to use the principal/keytab of a user you created if you have one. Hope this helps. ... View more

Hi @Jay See, The processor you are trying to use is using a Controller Service. You need to start this controller service to have your processor running correctly. Please go in the configuration of your processor: screen-shot-2017-02-21-at-92213-am.png Then click on the arrow at the right of the line of the controller service property to get to the page where you can define/enable your controller service: Ensure the controller service is correctly configured and enabled. Hope this helps. ... View more

Hi @Christophe Vico, Could you confirm, in NiFi, that the Ambari reporting task is correctly configured and running? You will find the information in the menu / Controllers / Reporting tasks. Thanks ... View more

Hi @Kaliyug Antagonist, NiFi provides a State API that allows you to do what you are looking for (using Zookeeper). Have a look here: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#state_manager There is a lot of existing processors using this state management and I believe this is the best way for you to understand how it works. For example all ListX processors should store a state to be sure they pick up at the right position in case of restart. Hope this helps. ... View more

Hi @Oliver Meyn, An option to resolve your issue is to define SANs (Subject Alternative Names) in your certificates so that each certificate has the FQDN as CN (as you did initially) but also the FQDN and the proxy/load balancer FQDN as DNS SAN. Have a look here - https://issues.apache.org/jira/browse/NIFI-3331 This will be possible with the TLS toolkit in the next release. This way you won't have any error regarding hostname mismatch. Hope this helps. ... View more