Thank you Matt sorry for the super delay response but it's very helpful ... View more

Correct, it is currently mounted on every node. I would have thought Nifi would dropped the file state once log files are deleted or archived but does not look like it. I could see 15K file states on the  tailing processor.  ... View more

@VJ_0082  Since your log is being generated on a remote server, You will need to use a processor that can remotely connect to the exteranl server to retrieve that log Possible designs: 1. The could incorporate a FetchSFTP processor in to your existing flow.  I assume your existing RouteOnAttriibute processor is checking for when an error happens with your script?  If so, add the FetchSFTP processor between this processor and your PutEmail processor.   Configured the FetchSFTP processor (configured with "Completion Strategy" of DELETE) fetch the specific log file created.  This dataflow assumes the log's filename is always the same. 2. This second flow could be built using the ListSFTP (configured with filename filter) --> FetchSFTP --> any processors you want to use to manipulate log --> PutEmail. The ListSFTP processor would be configured to execute on "primary" node and be configured with a "File Filter Regex".  When your 5 minute flow runs and if it encounters an exception resulting in the creation of the log file, this listSFTP processor will see that file and list it (0 byte FlowFile).  That FlowFile will have all the FlowFile attributes needed for the FetchSFTP processor (configured with "Completion Strategy" of DELETE) to fetch the log which is added to the content of the existing FlowFile.  If you do not need to extract from or modify that content, your next processor could just be the PutFile processor. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt   ... View more

Thank you!  I looked, but not in the right place.  I skimmed over the bucket API, but didn't look in detail as I assumed (incorrectly) that it was not relevant to updating a flow. ... View more

@SandyClouds  The ExecuteSQL processors do not support SSH tunnel.  The expectation by these processors is that the SQL server is listening on a port reachable on the network.  SSH tunnels are used to access the server via remotely and then execute a command locally on that SQL utilizing the SQL client on that destination server.   The ExecuteSQL processor uses a DBCPConnectionPool to facilitate the connection to the database.  The DBCPConnectionPool establishes a pool of connections used by one too many processors sharing this connection to execute their code.  A Validation Query is very import to make sure a connection from this pool is still good before being passed to requesting processor for use. While I have not done this myself, I suppose you could set up and SSH tunnel on each NiFi cluster server (example: https://linuxize.com/post/mysql-ssh-tunnel/).  Then you could still use the DBCPConnectionPool except use the established tunnel address and port in the database connection URL.   Downside to this is that NiFi has not control over that tunnel, so if the tunnel is closed, your dataflow will stop working until the tunnel is re-established.  The Validation Query will verify the connection is still good. If it is not, the DBCPConnectionPool will drop it and try to establish a new connection.   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Nifi-  You'll need to provide more detail around your use case in order to get more specific assistance. NiFi offers a number of processor components that can be used to ingest from a database: - ExecuteSQL  - ExecuteSQLRecord - CaptureChangeMySQL  <-- probably what you are looking for These ExecuteSQL processors will utilize a DBCPConnectionPool controller service for connecting to your specific Database of choice. SQL is what is needs to passed to these processors in order to fetch database table entries. The following processors are often used to generate the SQL in different ways needed by your use case to do this in an incremental fashion (for example: generating new SQL for new entries only so you are not fetching entire table over and over) - GenerateTableFetch - QueryDatabaseTable - ListDatabaseTable The CaptureChangeMySQL processor will output FlowFiles for each individual event.  You can then construct a dataflow to write these events to your choice of location.  That might be some other database.  Once you have your dataflow created for ingesting entries from your table in to NiFi, you'll need to use other processors within your dataflow for any routing or manipulation of that ingested data you may want to do before sending to a processor to write to the desired destination.  Possibly using  PutDatabaseRecord processor for example? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@MattWho  @ckumar  thanks for your inputs! I was able to resolve the issue following the steps you mentioned. Much appreciated! ... View more