@mridul_tripathi  That is not exactly the dataflow I was trying to convey, but good attempt. This is what I was envisioning: It start with fetching the of files from "SFTP1" using the listSFTP and FetchSFTP processors.  The ListSFTP processor will create a bunch of FlowFile attributes on the output FlowFile that can be used by the FetchSFTP to fetch the content and add it to the FlowFile.  In the FetchSFTP processor you will specify the SFTP1 Hostname, Username, and Password.  You will use NiFi Expression language to tell FetchSPT to fetch the specific content based in the FlowFile attributes created by ListSFTP: Next the FlowFile (now with its content from SFTP1) is passed to the CryptographicHashContent processor that will create a new FlowFile Attribute (content_SHA-256) on the flowFile with the content hash.  Unfortunately, we have no control over the FlowFile attribute name created by this processor.     Next The FlowFile is passed to an UpdateAttribute processor is used to move the (content_SHA-256) FlowFile to a new FlowFile attribute and remove the content_SHA-256 attribute completely so we can calculate it again later after fetch same file from SFTP2. I created a new FlowFile Attribute (SFTP1_hash) where I copied over the hash. Clicking the "+" will allow you to add a dynamic property.   Next I pass the FlowFile to ModifyBytes processor to remove the content from the FlowFile.   Now it is time to fetch the content for this same Filename from SFTP2 by using another FetchSFTP processor.  This FetchSFTP processor will be configured with the hostname for SFTP2, username for SFTP2, and password for SFT2.  We still want to use the filename from the FlowFile to make sure we are fetching the same file contents from SFTP2.  So you can still use "${path}/${filename}" assuming both SFTP1 and SFTP2 use the same path.  If not, you will need to set path manually (<some SFTP2 path>/${filename}).   Now you pass the FlowFile to another CryptographicHashContent processor which will have the content fetched from SFPT2 for the same filename.  At this point in time your FlowFile has a bunch of FlowFile attributes (including hash of both content from SFTP1 (SFTP1_hash) and SFTP2 (content_SHA256)and only the content from SFTP2.  So you'll pass it    Now it is time to compare those two hash attribute values to make sure they are identical using an RouteOnAttribute processor.  Here will create a NiFi Expression Language (NEL) expression to make this comparison.  Clicking the "+" will allow you to add a dynamic property.  Each Dynamic property added in this property becomes a new relationship on the processor.    ${content_SHA-256:equals(${SFTP1_hash})}   This NEL will return the value/string from FlowFile's "content_SH256" attribute and check to see if it is equal to the value/string from the FlowFile's "SFTP1_hash" attribute.  If true, the FlowFile will be routed to the new "Content-Match" relationship.  If false, it will be routed to the exiting "unmatched" relationship. Here you can decide if just want to auto-terminate the "Content-Match" relationship or do some further processing.  The Unmatched relationship will contain any FlowFiles where the content for two files of the same filename have content that did not match.  The FlowFile will contain the content from SFTP2. Hope this helps. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@MarinaM  Welcome to the Cloudera Community. Your Questions: Does NiFi allow segregation of data sources? - Would need a bit more information from you on this requirement/question.  You can certainly create non connected dataflows on the NiFi canvas for handling of NiFi FlowFiles from various sources to keep them separate.  However on the NiFi backend their is no segregation of content.  NiFi stores the content of 1 too many FlowFiles in content claims.  Anytime new content is created, it is written to the currently still open content claim no matter where in any of the dataflows that content is created. Content written to a content claim is immutable (can't be modified once written), so anywhere in yoru dataflow where you may make modification to a FlowFile's content would result in the new modified version of the content being written to a new content claim. Handling LEEF logs: logs reach NiFi with the original source IP but leave NiFi with NiFi’s source IP. - Would need details on how you have these logs arriving at NiFi and being ingested. Since QRadar first looks for the hostname in the payload (and if absent, uses the source IP), this could cause misidentification. - I am not familiar with QRadar, but perhaps you can modify the content when the hostname is missing in the payload via your NiFi dataflow(s)? Can NiFi be configured to retain the original source IP while forwarding logs, without modifying the original log (to comply with legal requirements)? - NiFi is a data agnostic tool and does not differentiate logs form any other content it is processing.  Content in NiFi is just bytes of data and it becomes the requirement of any individual processor that may need to interact with the content of the FlowFiles to understand the content format.  Would need to understand how you are ingesting these logs into NiFi. Some processor may be creating FlowFile attributes containing the source IP information which perhaps you can use later in your dataflow?  Perhaps another option is to build yoru dataflow to do lookups on the source IP and modify the syslog header when the hostname is missing?   Log Integrity & Authenticity: Does NiFi ensure log integrity and authenticity for legal and compliance purposes? - As mentioned for NiFi is data agnostic and content claims are immutable. Once a log is ingested the dataflow(s) you build can modify content if designed to do so and that modified log content is written to a new content claim.  Some Processors that modify content may create an entirely new FlowFile with that content referenced in it, but other may just modify the existing FlowFile to point and new modified content in the new content claim while keeping the original FlowFile identifier.  Typically this is the case with processor that have an "Original" relationship where the unmodified original FlowFile routes to this relationship while the modified content is assigned to an entirely new FlowFile which becomes a child FlowFile or that original.  LEEF Parsing: Is there a NiFi processor available to parse LEEF logs before storing them in HDFS? - Based on this IBM doc on LEEF (https://www.ibm.com/docs/en/SS42VS_DSM/pdf/b_Leef_format_guide.pdf), the LEEF logs consists of a RFC 5424 or RFC3164 formatted syslog headers which can be parsed by NiFi Syslog processors.  ListenSyslog ParseSyslog ParseSyslog5424 PutSyslog- Perhaps using PutSyslog instead of PutTCP can solve your Source IP issue you encounter by using PutTCP. There are also controller services that support these syslog formats: SyslogReader Syslog5424Reader Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@hus  Thank you for the clarification on your use case. The only purpose built processor NiFi has for appending lines to an exsisting file is the putSyslog processor. But it is designed to work with syslog formatted messages being sent to a syslog server for RFC5424 and RFC3164 formatted messages and can't be used to append directly to a local file.  However, your use case could be solved using the ExecuteStreamCommand processor and custom script.  The ExecuteStreamCommand processor passed a FlowFile's content to the input of the script. Example: I created the following script which I placed on my NiFi node somewhere where my NiFi service user has access.  Gave my NiFi service user execute permissions on the bash script (I named it file-append.sh) #!/bin/bash STD_IN=$(</dev/stdin) touch $1/$2 echo "$STD_IN" >> $1/$2 This script will take stdin from the executeStreamCommand processor which will contain the content of the FlowFile being processed.  $1 and $2 are command arguments i define in the ExecuteStreamCommand processor which I use to dynamically define the path and filename the content will be appended to .  It then takes FlowFiles content and either start a new file or append to a file if it already exists with the passed filename. You can see that I set my two command arguments by pulling values from the "path" and "filename" NiFi FlowFile attributes set on the FlowFile being processed. With this dataflow design you can append lines to various files as needed.   Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more