@New_User  There is the following resolved jira for the creation of a new putIceberg NiFi processor: https://issues.apache.org/jira/browse/NIFI-10442 Apache has size limits on distributions and it does not appear as though this processor nar was included in the Apache release. You would need to add the nifi-iceberg-processors-nar manually to your NiFi installation for it become available for use. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@tkchea  NiFi Remote Process Groups (RPG) transfer FlowFiles and not just the FlowFile content. So depending on the amount of metatdata/attributes on the FlowFile. the amount transferred would be larger. The RPG fetches Site-to-Site (S2S) details via a background thread the runs every 30 seconds regardless of existence of FlowFile.  These S2S details fetched will include details on the target NiFi (Number of nodes in target cluster, load on each node, RAW ports if configured, If HTTP is enabled, etc..).  These details are then used to facilitate the transfer of FlowFiles from client (RPG) and target NiFi (with Remote input or output ports).     The actual transfer of FlowFile will either happen over the HTTPS port (used by a lot of other transactions) or via a RAW socket port depending on configuration. Since a FlowFile consists of two parts (FlowFile Metadata and FlowFile Content), there is going to be disk and CPU I/O involved with writing to the flowfile_repository and content_repository.  So you may want to monitor those on both source and destination.  When it comes to the mutual TLS handshake, NiFi is not doing anything special here.  The client certificate presented is used to identify the client and verify authorization to the send to or pull from a remote port. You can also enable ssl handshake debug logging in the nifi bootstrap.conf file. java.arg.ssldebug=-Djavax.net.debug=ssl,handshake Of course you see all SSL handshakes including those when someone access the NiFi UI in the nifi-bootstrap.log file.   But this would allow you to see if you are seeing systematic slow TLS handshakes or only between these two networks.  You could also setup an RPG that sends to a remote input port on the same NiFi server.  The same TLS handshake will happen there as well.  Is it much faster (rules out an RPG issue.)  If it ends up being the network between NiFi servers, you'll need to investigate there perhaps using something like wireshark may help. Another test might involve using a postHTTP or InvokeHTTP to send to a ListenHTTP or HandleHTTPRequest processor on target server (can be setup to be secure or insecure using same keystore and truststore your NiFi's use). If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Girish007 Did you make sure that the NiFi directories and repository directories are excluded from any virus software scanning?  That is typically the external force that is likely to being making changes to these files. Do you have any other external software or processes scanning or access these directories? Thanks, Matt ... View more

@PurpleK  It is not clear what you mean when you say "Files that are in the 500GB+ range are taking several hours to move onto the unpack stage.". So FlowFile(s) are released to a downstream connection until processing of the source file is complete.  The source file will still be represented in the queued count of the connection feeding a processors even while that processor is executing on that FlowFile. When you moving on to unpack stage, are you referring to some upstream processor feeding the connection to the UnpackContent processor taking awhile to queue some FlowFile on that downstream connection, or are you referring to once the file is queued it take awhile for unpack to complete execution on it creating on the unpacked FlowFiles and then remove original zip from upstream connection queue? Step 1 is identify the exact place(s) it is slow. Adding additional concurrent tasks to a processor has no impact on speeding up the execution on a specific source FlowFile.  1 thread get assigned to each execution of the processor and in the case of unpackContent, each tread executes against 1 FlowFile from upstream connection.  Adding multiple concurrent tasks will allow multiple upstream FlowFiles to be processed concurrently.  IMPORTANT: Increment concurrent tasks slowly while monitoring CPU load averages.  Adding too many concurrent tasks on any one processor can impact other processors in your dataflow Event Driven Processor scheduling strategy is deprecated and will eventually go away (mostly like next major release) and should not be used.  So increasing the Max Event driven Thread count under controller settings will have no impact unless you are using that strategy in your flow.  It does create event threads, but they would not consume CPU if you are not using event driven scheduling anywhere in your dataflow(s). NiFi is a data agnostic service, meaning it can handle any stat type in its raw binary format.  NiFi can do this because it wraps that binary content in a NiFi FlowFile.  A NiFi FlowFile is what you see moving form processor to processor in your dataflows and int becomes the responsibility of the processor to understand the FlowFile's content should it need to read it.  I bring this up because a FlowFile adds a small bit of overhead as it has to generate FlowFile metadata for every FlowFile created. When it comes to your 500GB+ zip files... 1. Do they consist of many small and/or large files? NiFi must create a FlowFile for each file that results from unpacking the original zip. 2. Do you see a lot of Java Garbage Collection (GC) pauses happening?  All GC is stop the world.  GC is normal operation or any JVM, but if GC is happening very often it can impact flow performance with constant pauses due to stop the world nature of GC.  The larger the JVM memory that longer the stop the world event will be. 3. Any exceptions in your niif-app.log?   You may also find this article helpful, it is old but majority of guidance is still very valid.  Latest NiFi version support Java 8 and Java11, so you can ignore the G1GC recommendations if your are using Java 11. https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999   Hopefully the concurrent tasks on your processor(s) excuting against the content of lareg FlowFiles will help you better utilize your hardware and achieve overall better throughput.  Keep in mind that it only allows concurrent execution on multiple source FlowFiles, so will not improve speed at which a single FlowFile will be processed by a given processor. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

Not ruling out something environmental here, but what is being observed is validation working and processor execution not while both those processes should be using the same basic code. The 3 loggers that would produce Debug logging output suggested in my previous post may shed more light on the difference in the output logging when validation is done versus running (starting) the processor.  So that is probably the best place to start. ... View more

@lben  if you saw a bulletin on the processor reporting a failure in execution, that should also be in the nifi-app.log. You can also modify the logback.xml to change the log level of NiFi or even just the ListSFTP processor class to hopefully capture more detail on the failure. Does SFTP to target server work from command line as the NiFi service user?  SFTP is just FTP over SSH.  But yes, SFTP servers can be configured to only allow SFTP connections. So to get more logging out of the listSFTP processor class you could add these loggers the area where all the other loggers start to show up in the NiFi logback.xml:   <logger name="org.apache.nifi.processors.standard.ListSFTP" level="DEBUG"/>   <logger name="net.schmizz.sshj" level="DEBUG"/> <logger name="com.hierynomus.sshj" level="DEBUG" /> Thanks, Matt ... View more