Keep in mind that a Docker image is immutable and when any changes are made, this is both a new layer and image. To save the templates in your current approach, you would need to save the Docker image via a docker commit. Although this would not be maintainable, nor would it be best practice. Instead, you would likely want to make use of Docker volumes (https://docs.docker.com/engine/tutorials/dockervolumes/) such that the data could be persistent on the host. In this case, you could use a folder on the host that contains your seed templates that would be mapped into running instances of the image you created. Something to the effect of: docker run -it -v /<host templates dir>/:/opt/nifi/conf/templates/ my_image This will allow you to provide templates to your container and also capture those new ones introduced in the image on the host system. This set of cached templates on your host will also enable you to then mount these templates to multiple instances that would be colocated on that host. ... View more

The following is under the assumption you do not explicitly require using Site to Site to transmit the data. While there are facilities that could support this, it is not an extension point and currently does not provide control over how FlowFiles are delivered. The simplest and most naive approach would be to use RouteOnAttribute to route each FlowFile to a given relationship and then use that to feed your transmission processor of choice. In this case, a PostHTTP sending to a ListenHTTP would be one way of attack that would allow transmission formatted as FlowFiles. Depending on the source system you might, such as if it was clustered, need to use expression language to additionally mark the destination system and use that to dynamically craft the resultant POST URL to the associated listener. This is fairly static and simple but would cover the use case you were anticipating. ... View more

There is not such a mechanism in place for ExecuteStreamCommand. ExecuteProcess does have the ability to batch but based on your mention of incoming event rate, seems like the need for handling input which ExecuteProcess does not provide. Depending on the nature of your parsing/process, it may be possible to convert this to use InvokeScriptedProcessor which could tie said parser/process to the component lifecycle. ... View more

Your best strategy for this case would be to leverage Expression Language [1] in the Remote URL field of your InvokeHTTP. You could then use a GetFile to take your set of URLs from a file on the filesystem or even an InovkeHTTP to another location to introduce your list of URLs scheduling strategy of Cron to pull in that list as desired. This list could then be split into separate events using a SplitText. From here, we can promote the contents of each of these splits to an attribute (perhaps target.url) using ExtractText. This would then be passed to your InvokeHTTP which would make use of the target.url attribute by specifying ${target.url} in the "Remote URL" field mentioned above. [1]https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html ... View more

With MergeContent, it is possible to specify a Max Bin Age that will prevent a data starvation condition that prevents the latest data from being held in limbo. Accordingly, you can make a best effort to get an appropriately sized file to place in HDFS but not at the cost of data being held indefinitely. ... View more

If you are referring to a batch or PowerShell script, this can be accomplished via ExecuteProcess [1] or ExecuteStreamCommand [2]. Both will allow you to execute processes on your host environment and collect its output to stdout from the execution in a resultant FlowFile for further processing. ExecuteStreamCommand additionally allows you to take an incoming FlowFile and pipe its content across stdin to the associated process. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.ExecuteProcess/    [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteStreamCommand/index.html  ... View more

NiFi cannot currently do this with the provided *FTP* implementations. However, the commons-net library that is used for FTP also provides an FTPSClient as well that would be quite similar in structure as well as providing the needed certificates (likely through an SSLContextControllerService) and related properties. ... View more

Clustering is bidirectional in that there is a heartbeat mechanism as you are seeing across the protocol port you have configured and then the NCM replicating requests and communicates to the nodes via their REST API, configured across the same host and port that the UI for each would be available on. This information is relayed from each node to the manager as part of the joining process. The primary properties beyond those for the clustering protocol are: nifi.web.http.host nifi.web.http.port which would need to be accessible from the NCM. With the request that results in the "... No nodes were able to process this request", there should additionally be a stacktrace on the NCM that outputs the address(es) that it is anticipating to be available. Verify connectivity to those sockets from your NCM. If nifi.web.http.host is not explicitly set, this will default to localhost which then be interpreted by the manager incorrectly when transmitted with the heartbeat. Beyond that, if that does not turn up any additional paths, if you are able to share your NCM and one of your node's web and clustering properties it may help to debug a bit further. ... View more

@surender nath reddy kudumula There are several other processors that handle various types of data formats and protocols and always more in development as we continue to grow the product and community. Some common items include UDP and Syslog listeners. I think there are certainly some additional items that we can add into NiFi to facilitate a use case such as streaming HTTP and actually started creating a ticket to provide some functionalities to at least cover general cases (NIFI-1389) Feel free to add any suggestions toward that functionality. Thanks! ... View more