@syntax_  NiFi does not provide a method for uploading templates in bulk.   But anything you can do via the UI, you can also do via rest-api calls [1] through curl.  So you could script the bulk upload through rest-api external to NiFi's UI. I would strongly discourage bulk uploading templates to NiFi.  Templates should be uploaded as needed, instantiate to the canvas and then template deleted from NiFi.   All uploaded templates even if not instantiated to the canvas become part of the flow.xml loaded in to heap memory.  So keeping a large number of templates uploaded to your NiFi can have a considerable impact on JVM heap memory usage because now you have not only your active flow on the canvas in memory, but also these templates. Additionally, In a NiFi cluster on startup the flow.xml.gz is uncompressed, loaded in to heap memory, and a flow fingerprint created on every node.  These flow fingerprints are then compared to make sure all nodes joining the cluster are running with the same flow.xml.  Since these templates can just add unnecessary size to the flow.xml.gz, this can impact startup times and flow fingerprint comparison time. A better approach is to get your reusable flow migrated into NiFi-Registry [2].  You can have 1 to many NiFi's all connect to a single NiFi-Registry where all your flows exist.  Then all you need to do is drag a process group to the canvas and select "import" which will allow you to select one of these flows from NiFi-Registry and add it to your canvas.  Now you have your flow without the additional impact on heap from having it in NiFi both as a template and as a dataflow on your canvas with easy access to load same flow over and over.  I encourage you to explore NiFi-Registry. [1] http://nifi.apache.org/docs/nifi-docs/rest-api/index.html [2] https://nifi.apache.org/registry.html   If you found this response helped with your query, please take a moment to login in and click accept on this solution. Thanks, Matt ... View more

@robnew    The "CompressContent" [1] processor can be used to decompress gz files. My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.    [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.CompressContent/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.IdentifyMimeType/index.html [3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you found this helpful with your question, please take a moment to login and click accept on this solution. Thanks, Matt ... View more

@Dmitry  Trying to fully understand your use case here... You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute "manager_name" with the same value? If so, the MergeContent [1] or MergeRecord [2] processors could do that for you.  You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile.  Those bins are then merged based on criteria set in other properties on the processor. The merged output FlowFile will have several new attribute written to it.  One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.   You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeContent/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeRecord/index.html   If you found this information helped with you question, please take a moment to login and click accept on this solution. Matt ... View more

@Justee  So with NiFi, there is a UI, but you need to make sure there is a route to the host (where you are laughing your browser) and target: <hostname/IP>:<port>  If your NiFi in startup is binding to private network interface or only the loopback IP address, you would most likely be unable to reach those IPs from your client machine on a public network. From your client machine, can you ping that "<hostname/IP>"? Does the server where your NiFi is running have multiple network interfaces for perhaps a public and a private network?  Did you try changing the IP that NiFi binds to in nifi.properties file: nifi.web.http.host= Did you try to tell your NiFi to bind to multiple network interfaces in the nifi.properties file if they exist in that server?  For example: nifi.web.http.network.interface.eth0=eth0 nifi.web.http.network.interface.eth1=eth1 Your servers network interfaces may be different then above example. If you found this solution helpful, please take a moment to login in and click accept on all solutions that helped you. Thanks, Matt ... View more

@pherschmann    The "nifi is running but not responding to ping requests" is not referring to ability to ping the host. When NiFi is started, you are starting the NiFi bootstrap process. This bootstrap process then starts a child NiFi process which is what runs the main NiFi that you interact with.  When you run the status command, it us asking the bootstrap process to return the status of that child process.  It is telling you that it sees that the PID for that child process still exists so reports it is running, but when it tries to get a response from that child process it fails and logs that the status ping got no response. This almost always is the result of some resource contention within NiFi.  The NiFi process may have been going through Java Garbage Collection (GC) at the time you ran the status command. (all GC is stop-the-world event, so JVM will not respond to anything else), or thread usage (CPU consumption) was high at the time, etc... If you see this allot or are seeing node disconnection or UI stability issues, you may want to start monitoring resources on the host, or maybe enable GC logging in your NiFi to see if the service is spending a lot of time doing GC.  Increasing heap allocated to NiFi or making dataflow design changes to reduce the heap memory footprint of your current dataflow(s) design would help there. Hope you found this information helpful. If so, please take a moment to login and click accept on this solution, Matt ... View more