GetFile will delete the processed file by default. Check the Keep Source File setting, for example. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetFile/index.html A better alternative for processing files without touching them is a combination of ListFile/FetchFile processors. These will maintain the state internally in NiFi to track the processed files. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html ... View more

Hi Jobin, The backpressure should be treated as a threshold, not an absolute barrier. Due to scheduling semantics and some processors like e.g. UpdateAttribute doing internal micro-batching, the number of objects (or size) can go over the set limit. It will still engage the backpressure and will not turn it off until the level falls below the threshold, but in general you shouldn't worry too much about the number going slightly above. This is merely due to internal implementation details. ... View more

Setting a remote file to /home/temp/ is wrong. It expects a filename, not a directory. A combination of ListSFTP and FetchSFTP will ensure the filenames are passed to the FestSFTP via a known attribute name. Here's an example template from the gallery: https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_template.xml ... View more

Yes, use MonitorActivity processor: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MonitorActivity/index.html ... View more

One needs to allow for a reflect function, it's blacklisted by default. See e.g. this discussion: https://community.hortonworks.com/questions/25828/udf-reflect-is-not-allowed-beeline.html ... View more

ExecuteProcess doesn't take input. It is a source of data (e.g. from that process). What you're looking for is ExecuteStreamCommand, which allows for passing in a flowfile and evaluate attributes dynamically: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteStreamCommand/index.html ... View more

David, it's here: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.0/bk_dataflow-command-line-installation/content/hdf_isg_hardware.html Although I don't think guidelines have changed. ... View more

Have you tried adding this 'set' statement as the first line of the query, terminated by a semicolon with newline? Next, try quoting the column name, too. ... View more

It doesn't affect scheduling, it's a normal (failure) path for your data. When one auto-terminates a relationship it means that a FlowFile which was routed there is dropped and finishes its life in the flow (but still remains for some time in provenance/content repositories for history). ... View more

Site-to-site is much more versatile than multi-DC communications (though this is a great use case for this NiFi's feature). S2s can link multiple clusters (or standalone instances), can even connect a cluster to itself (for data re-distribution), as well as used for MiNiFi to NiFi communication. It's also bi-directional, meaning it can be push/pull in either direction. At the end of the day this means you will be able to communicate e.g. over a corporate HTTP proxy regardless of the inbound/outbound firewall rules, there's enough flexibility to accommodate these scenarios in s2s. Read up more here https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site ... View more

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale. ... View more

Hi, the nifi-api-deploy doesn't have support for NiFi 1.x API. ... View more

Hi Avijeet. This is currently not possible, but do take a look at the following proposal to understand where things are going in the future: https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows ... View more

Please check in the nifi-app.log file, not bootstrap. Most probably there is a service in your host already taking the 8080 port. You can edit conf/nifi.properties and modify the port to a non-conflicting one. ... View more

Let's simplify the config to eliminate misconfiguration in other areas. If you want A to initiate the connection, you can remove those nifi.remote.* properties on the A side. Next, the port 8443 is not the best choice, as it often is used for https on non-privileged port. It's not even required now, as NiFi's s2s can tunnel over HTTP. Remove input.socket.port on the B side. Finally, please share which URL you used in the RPG setup (the UI on the A side). The GoTo action simply opens a new browser window. What we are looking for is being able to Refresh the RPG and see ports after 20-30 seconds after initial setup. Few more things to validate. I understand this is not a secured instance, so port access permissions aren't in play. Make sure, though, that instance B has input/output ports added the root top-level processing group, this is a requirements for the s2s. ... View more

Hi Obaid, currently HDF installs its own ZK quorum. In the future colocating HDF & HDP in the same cluster will be supported through a single Ambari instance to better reuse infrastructure components. http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_ambari-installation/content/ch_installing-ambari.html ... View more

Hey Andrew, If your message is a valid JSON (it seems to be), then a parser is able to read it an decode a literal string. The next trick is to put this results into a FlowFile content/body and run another EvaluateJsonPath/JoltTransformer chain. I did a quick experiment here, and it seems to work fine. ... View more

Scott, Use ExecuteScript processor, as mentioned in the article. Looks like you are putting an embedded script into ExecuteProcess, which is meant to invoke OS shell commands, hence the error. ... View more