@mridul_tripathi  That is not exactly the dataflow I was trying to convey, but good attempt. This is what I was envisioning: It start with fetching the of files from "SFTP1" using the listSFTP and FetchSFTP processors.  The ListSFTP processor will create a bunch of FlowFile attributes on the output FlowFile that can be used by the FetchSFTP to fetch the content and add it to the FlowFile.  In the FetchSFTP processor you will specify the SFTP1 Hostname, Username, and Password.  You will use NiFi Expression language to tell FetchSPT to fetch the specific content based in the FlowFile attributes created by ListSFTP: Next the FlowFile (now with its content from SFTP1) is passed to the CryptographicHashContent processor that will create a new FlowFile Attribute (content_SHA-256) on the flowFile with the content hash.  Unfortunately, we have no control over the FlowFile attribute name created by this processor.     Next The FlowFile is passed to an UpdateAttribute processor is used to move the (content_SHA-256) FlowFile to a new FlowFile attribute and remove the content_SHA-256 attribute completely so we can calculate it again later after fetch same file from SFTP2. I created a new FlowFile Attribute (SFTP1_hash) where I copied over the hash. Clicking the "+" will allow you to add a dynamic property.   Next I pass the FlowFile to ModifyBytes processor to remove the content from the FlowFile.   Now it is time to fetch the content for this same Filename from SFTP2 by using another FetchSFTP processor.  This FetchSFTP processor will be configured with the hostname for SFTP2, username for SFTP2, and password for SFT2.  We still want to use the filename from the FlowFile to make sure we are fetching the same file contents from SFTP2.  So you can still use "${path}/${filename}" assuming both SFTP1 and SFTP2 use the same path.  If not, you will need to set path manually (<some SFTP2 path>/${filename}).   Now you pass the FlowFile to another CryptographicHashContent processor which will have the content fetched from SFPT2 for the same filename.  At this point in time your FlowFile has a bunch of FlowFile attributes (including hash of both content from SFTP1 (SFTP1_hash) and SFTP2 (content_SHA256)and only the content from SFTP2.  So you'll pass it    Now it is time to compare those two hash attribute values to make sure they are identical using an RouteOnAttribute processor.  Here will create a NiFi Expression Language (NEL) expression to make this comparison.  Clicking the "+" will allow you to add a dynamic property.  Each Dynamic property added in this property becomes a new relationship on the processor.    ${content_SHA-256:equals(${SFTP1_hash})}   This NEL will return the value/string from FlowFile's "content_SH256" attribute and check to see if it is equal to the value/string from the FlowFile's "SFTP1_hash" attribute.  If true, the FlowFile will be routed to the new "Content-Match" relationship.  If false, it will be routed to the exiting "unmatched" relationship. Here you can decide if just want to auto-terminate the "Content-Match" relationship or do some further processing.  The Unmatched relationship will contain any FlowFiles where the content for two files of the same filename have content that did not match.  The FlowFile will contain the content from SFTP2. Hope this helps. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@MarinaM  Welcome to the Cloudera Community. Your Questions: Does NiFi allow segregation of data sources? - Would need a bit more information from you on this requirement/question.  You can certainly create non connected dataflows on the NiFi canvas for handling of NiFi FlowFiles from various sources to keep them separate.  However on the NiFi backend their is no segregation of content.  NiFi stores the content of 1 too many FlowFiles in content claims.  Anytime new content is created, it is written to the currently still open content claim no matter where in any of the dataflows that content is created. Content written to a content claim is immutable (can't be modified once written), so anywhere in yoru dataflow where you may make modification to a FlowFile's content would result in the new modified version of the content being written to a new content claim. Handling LEEF logs: logs reach NiFi with the original source IP but leave NiFi with NiFi’s source IP. - Would need details on how you have these logs arriving at NiFi and being ingested. Since QRadar first looks for the hostname in the payload (and if absent, uses the source IP), this could cause misidentification. - I am not familiar with QRadar, but perhaps you can modify the content when the hostname is missing in the payload via your NiFi dataflow(s)? Can NiFi be configured to retain the original source IP while forwarding logs, without modifying the original log (to comply with legal requirements)? - NiFi is a data agnostic tool and does not differentiate logs form any other content it is processing.  Content in NiFi is just bytes of data and it becomes the requirement of any individual processor that may need to interact with the content of the FlowFiles to understand the content format.  Would need to understand how you are ingesting these logs into NiFi. Some processor may be creating FlowFile attributes containing the source IP information which perhaps you can use later in your dataflow?  Perhaps another option is to build yoru dataflow to do lookups on the source IP and modify the syslog header when the hostname is missing?   Log Integrity & Authenticity: Does NiFi ensure log integrity and authenticity for legal and compliance purposes? - As mentioned for NiFi is data agnostic and content claims are immutable. Once a log is ingested the dataflow(s) you build can modify content if designed to do so and that modified log content is written to a new content claim.  Some Processors that modify content may create an entirely new FlowFile with that content referenced in it, but other may just modify the existing FlowFile to point and new modified content in the new content claim while keeping the original FlowFile identifier.  Typically this is the case with processor that have an "Original" relationship where the unmodified original FlowFile routes to this relationship while the modified content is assigned to an entirely new FlowFile which becomes a child FlowFile or that original.  LEEF Parsing: Is there a NiFi processor available to parse LEEF logs before storing them in HDFS? - Based on this IBM doc on LEEF (https://www.ibm.com/docs/en/SS42VS_DSM/pdf/b_Leef_format_guide.pdf), the LEEF logs consists of a RFC 5424 or RFC3164 formatted syslog headers which can be parsed by NiFi Syslog processors.  ListenSyslog ParseSyslog ParseSyslog5424 PutSyslog- Perhaps using PutSyslog instead of PutTCP can solve your Source IP issue you encounter by using PutTCP. There are also controller services that support these syslog formats: SyslogReader Syslog5424Reader Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thanks for the reply, however I am delighted to say that I have solved the issue. I had to reinstall the image and change the custom port to a proper empty one. On some sites 111 appears to work, which is the default setting, but in my case it wasn't for custom16. I had to change it before running the image because changing it later has no effect! Thanks @VidyaSargur for your prompt reply. Have a good one ... View more

It appears that the issue is not appearing when using docker (I was using podman before).  I'm not sure how it could affect the comms with the agents, but I've had issues on previous versions related to the DNS resolution of the nodes returning multiple names that do not necessarily correspond to the hostname configured in /etc/hostname I will try to confirm this and update this thread for anyone trying the same, but I'll mark this as solved since I can confirm the issue is related to using podman. ... View more