@oka  I believe your issue is that "-" is not a valid character in the JMS specification. https://stackoverflow.com/questions/30022453/how-to-set-content-type-header-for-jms-message While the "-" is valid for AMQP messages. There is a ConsumeAMQP processor but I see it only supports AMQP version 0.91 You don't have a issue connecting but just sound like these specific headers are ignored since they contain that invalid JMS character. There is an existing Apache NiFi jira (https://issues.apache.org/jira/browse/NIFI-14670) for adding AMQP 1.0 support to the AMQP processors; however, it is open and unassigned. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fnimi   I am not clear on "It dies for 15-30 minutes".    What version of Apache NiFi are you using? Is this a standalone instance of NiFi or a multi-node NiFi Cluster? Does the NiFi child process die and get launched agin by the NiFi bootstrap process?  Any exceptions thrown in the nifi-app.log or nifi-bootstrap.log (timeouts, OutOfMemory(OOM), etc)?  Are you seeing a long stop-the-world Java Garbage Collection event? How large are these "big" process groups? The more detail you can provide, the better chance a community member will be able to provide some suggestions. Thank you, Matt ... View more

@nisaar  I see you set retry on the ListSMB processor "success" relationship.  You should never set retry on any "Success" relationship.  The connection retry logic is triggered when a FlowFile would be routed to the relationship.  Instead of routing the FlowFile to the relationship it remains on the upstream connection to be "retried".  That means that only when FetchSMB successfully lists a file will it trigger retry.  Plus retry implies a there is some upstream location to persist a source FlowFile. Only once all retry attempts are exhausted will the FlowFile actually be placed on the connection containing the "success" relationship. I would expect unexpected behavior with such a configuration.  Using "retry" on a failure or other error type relationship makes sense since these are the relationships you don't expect FlowFiles to be routed to.  so makes sense on your FetchSMB processor so that source FlowFile will remain on the inbound connection to FetchSMB until both the configured retry attempts are made.  I both retries fail I see you terminate the "Failure" relationship which effectively throws away that file.   I don't know that terminate is what i would do here either since ListSMB was successful in order to produce FlowFile waiting at FetchSMB.  Plus state is recorded by ListSMB and the file that failed ot be fetched would never get listed again unless you touched file to update last modified time or cleared state on listSMB processor.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  What version of Apache NIFi is being used? How is your PutSFPT processor configured? Can you share the complete stack trace from the nifi-app.log? Matt ... View more

 @nisaar  Just so I am clear in your dataflow setup... - ListSMB  processor configured to use cron scheduling (get scheduled for ru every ~30 mins) - ListSMB "success relationship" routed via a connection to FetchSMB processor - FetchSMB configured to use Timer Driven scheduling with:   - Flow looks like this:   The "retry" when set on a relationship controls whether a FlowFile remains on the inbound connection to the processor or gets routed immediately to the destination relationship. The number of retries is how many attempts will be made to reprocess the source FlowFile before finally routing to the destination relationship.  You have 2 so that FlowFile will get terminated if it is not successfully fetched after 2 failed attempts. I am not clear on this statement you made: On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully. There is no "retry" on the listSMB.  It simply get scheduled to run at the configured run schedule and listed based upon previous stored state and processor property configuration. 1. How are files being added to the source SMB directory? 2. As files are added how is the the last modified timestamp being updated?  (if they are being moved to the SMB as an atomic move, the timestamp on the file may not change which can result in ignored files because another file already listed resulted in sate holding a more recent timestamp). 3. What "Listing strategy" are you using?  I recommend using "Tracking Entities" to avoid issue from question 2 above. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  When you execute the SFTP CLI, are you doing so as the same user that owns the running NiFi process?  If not, switch to teh NiFi service user and then try your SFTP CLI command again to see if it is successful or if it ask you to accept the host key for the SFTP target.  If you are asked to accept the host key, try running your NiFi putSFTP processor again after you accept the host key on the NiFi service user account. It does not matter what user is authenticated in NiFi or which user created the dataflow on the NiFi canvas.  All components are executed as the NiFi service user.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@RohanBajaj  This is a very old thread with a very old version of Apache NiFi from the very early days of NiFi's introduction of the load balancing capability through the connections.  I recommend you start a new community question with the specifics of any issue you are having to get the best possible assistance from the community members. Thank you, Matt ... View more

@donaldo71  I have not been able to identify a known issue that aligns with the description you have shared. That is an interesting sequence of events on a single FlowFile (SEND followed by clones).  Can you share the "Relationships" configuration of your putSQL processor? Make sure you have not checked the "retry" box on the "success" relationship.   Something you might want to try to see if same issue persists is to check the box for "retry"  on the retry relationship.   This allows the original FlowFile to remain in the inbound connection up to the configured number of retry attempts (default 10) before being routed to retry relationship.  I'd be curious of your observations post the above configuration change. Would you be willing to download the your flow definition json for this dataflow and share it? This is your full "SQL Statement" set in your ptSQL processor? UPDATE tbl SET status = 'proceed', startDate = GETDATE() WHERE messageId = ${messageId} Where are your utilizing those two attributes that go missing? can you share your UpdateAttribute processor configuration? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@donaldo71  It would be helpful to know the full Apache NiFi version being used so I can see if there are any known issue that align here. I assume that you have a prepared SQL statement configured in your PutSQL processor that uses NiFi expression language to insert values from the Source FlowFile's attributes? Are you saying that on the very first failed execution of a "retry" routed FlowFile (the attempt that still had the "formatDateFrom" attribute set, there was no exception different from all subsequent retry requests made with FlowFile now missing that attribute)? Thank you, Matt ... View more

@donaldo71 I am trying to follow your flow description here and am not very clear.  Can you share your flow definition and indicate where you are seeing the failure.  What version of Apache NiFi are you using? You have an UpdateAttribute processor that has its success relationship routed via a connection to the PutSQL processor.  How is putSQL configured? When "something is wrong", what exception is being thrown in the nifi-app.log? The more detail you can provide the better chance. community member may be able to provide guidance and suggestions. Thank you, Matt ... View more