GetFile will delete the processed file by default. Check the Keep Source File setting, for example. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetFile/index.html A better alternative for processing files without touching them is a combination of ListFile/FetchFile processors. These will maintain the state internally in NiFi to track the processed files. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html ... View more

Hi Jobin, The backpressure should be treated as a threshold, not an absolute barrier. Due to scheduling semantics and some processors like e.g. UpdateAttribute doing internal micro-batching, the number of objects (or size) can go over the set limit. It will still engage the backpressure and will not turn it off until the level falls below the threshold, but in general you shouldn't worry too much about the number going slightly above. This is merely due to internal implementation details. ... View more

Setting a remote file to /home/temp/ is wrong. It expects a filename, not a directory. A combination of ListSFTP and FetchSFTP will ensure the filenames are passed to the FestSFTP via a known attribute name. Here's an example template from the gallery: https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_template.xml ... View more

Yes, use MonitorActivity processor: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MonitorActivity/index.html ... View more

One needs to allow for a reflect function, it's blacklisted by default. See e.g. this discussion: https://community.hortonworks.com/questions/25828/udf-reflect-is-not-allowed-beeline.html ... View more

ExecuteProcess doesn't take input. It is a source of data (e.g. from that process). What you're looking for is ExecuteStreamCommand, which allows for passing in a flowfile and evaluate attributes dynamically: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteStreamCommand/index.html ... View more

David, it's here: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.0/bk_dataflow-command-line-installation/content/hdf_isg_hardware.html Although I don't think guidelines have changed. ... View more

Have you tried adding this 'set' statement as the first line of the query, terminated by a semicolon with newline? Next, try quoting the column name, too. ... View more

It doesn't affect scheduling, it's a normal (failure) path for your data. When one auto-terminates a relationship it means that a FlowFile which was routed there is dropped and finishes its life in the flow (but still remains for some time in provenance/content repositories for history). ... View more

Site-to-site is much more versatile than multi-DC communications (though this is a great use case for this NiFi's feature). S2s can link multiple clusters (or standalone instances), can even connect a cluster to itself (for data re-distribution), as well as used for MiNiFi to NiFi communication. It's also bi-directional, meaning it can be push/pull in either direction. At the end of the day this means you will be able to communicate e.g. over a corporate HTTP proxy regardless of the inbound/outbound firewall rules, there's enough flexibility to accommodate these scenarios in s2s. Read up more here https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site ... View more

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale. ... View more

Hi, the nifi-api-deploy doesn't have support for NiFi 1.x API. ... View more