@jame1997  Not much to look at from a NiFi side. NiFi is writing to the network successfully and there is some loss then happening between NiFi and your syslog server.   Resource usage affecting your NiFi would only slow down processing but not result in dataloss within NiFi.  So the PutSyslog would successfully write all bytes to the network before passing the FlowFile to the "success" relationship.  Using TCP of course would allow NiFi to confirm successful delivery thus allowing NiFi to appropriately retry, or route to either failure or success relationships. You could look at the data rate NiFi is writing from the putSyslog by looking at the stats on the processor.  Then maybe you could experiment with: 1. netstat - nsu to check for UDP packet loss.   2. using a network monitoring tool Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@s198 NiFi has no ability to merge files remotely.  NiFi would need to consume all the files (ListHDFS --> FetchHDFS), then merge the content of those FlowFiles (MergeContent or MergeRecord), then use UpdateAttribute to set desired filename on merged file, and finally write the merged file back to HDFS using PutHDFS processor. If you are using a NiFi Cluster, you would need to do all this merging on one node of the cluster, NiFi nodes can only execute against the FlowFiles present on that one specific node. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@TreantProtector  Everything the user adds to the canvas including controller service and reporting tasks are auto-saved in the flow.json.gz.  Each time a change is made the current flow.json.gz is archived and new flow.json.gz is generated.   Within the flow.json.g are all components (processors, connections, controller services, reporting tasks, funnels, process groups, ports, parameters, etc.)  and their configurations.  Any configuration property that is "sensitive" (passwords) are going to be encrypted in the flow.json.gz file.  So in order to load that flow.json.gz in another NiFi, you would need to know the nifi.sensitive.props.algorithm and nifi.sensitive.props.key used by the original NiFi which it came from. Encrypted Passwords in Flows If you don't have that info, the flow.json.gz can still be loaded on another NiFi after manually editing the file to remove all the "enc{...}" values.  Once flow.json.gz loads, an authorized user would need to re-enter all passwords in all components where it is needed via the NiFi UI. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@s198  The List<abc> type processor are source based processors that do not accept inbound connections since they are designed to create FlowFiles and designed to modify existing FlowFiles. I am not clear on what "So we used Sqoop completion" does to create a FlowFile in your NiFi dataflow which is then passed to RouteOnAttribute (assuming this is processor you are referring to by "Router Attribute") via a connection.   What Attributes exist on the FlowFile being processed by the RouteOnAttribute  processor.  Any FlowFile attributes on this FlowFile about the specific file needing to fetched by the FetchHDFS processor (like filename and path)? ----- If sqoop job output produced 1 FlowFile for each HDFS file to be fetched and each of those FlowFiles has attributes for path and filename of the HDFS file content to be fetched, you could do following: Set the default NiFi expression language statement  "${path}/${filename}" in the "HDFS File Name" Property of the FetchHDFS processor.  Those two FlowFile attributes are expected to be in the format:         Attribute Name:  Attribute value: filename The name of the file that will be read from HDFS. path The path is set to the absolute path of the file's directory on HDFS. For example,  "/tmp/abc/1/2/3".  Attribute names are case sensitive. ----- If the sqoop job simply outputs 1 FlowFile from which you expect to fetch a lot of HDFS files, that is not how FetchHDFS functions.  FetchHDFS expects one FlowFile for each HDFS file content being fetched.  FetchHDFS does create new FlowFiles, it only adds content to an existing FlowFile. If this matches your scenario, you may be able to use the GetHDFSFileInfo processor that does accept and inbound connection.  It can be configured with just a path.  If you set "Group Results = None" and "Destination = Attributes", you could send the produced FlowFiles to FetchHDFS to get the content for each FlowFile output. You would still need your RouteOnAttribute processor to make sure only FlowFiles where "${hdfs.type} = file" were routed to FetchHDFS and others types are discarded.  You would probably also want an UpdateAttribute processor so you could set the filename of the FlowFile to the hdfs.objectName (done by adding dynamic property filename = ${hdfs.objectName}). Then feed those FlowFiles to your FetchHDFS processor configured to use the ${hdfs.path}${hdfs.objectName} NiFi Expression statement in the "HDFS File Name" Property. ------ If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more