You are awesome @SAMSAL Thanks so much for the great information.  This helps me a lot.  Thank goodness for this community, because I think the documentation is nearly adequate. ... View more

@Jisson  I don't see ExecuteStream in the thread dump provided.   Let's clarify first what you mean by "stuck"... When the processor is in this "stuck" state, does the processor indicate that it has an active thread?  A NiFi processor will show a small number in its upper right corner when it has an active thread(s).  Below example shows an ExecuteStreamCommand processor with "1" active thread: If your processor has no active threads, it is not stuck/hung.  It is simply does not have a thread to execute the command.  This could happen if all thread from the max timer driven thread pool in NiFi are already being used by other components.  We would call this a thread starved processor.  If your CPU load average is good, you could increase the size of the thread pool to see if that helps.  NiFi out-of-the-box sets the "Maximum Timer Driven Thread Count" Pool to 10.  You can change this from the NiFi Ui --> global menu (upper right corner) --> Controller Settings --> General tab. If your processor does show an active threads, i'd expect to see that thread in the thread dump.  Also keep in mind that a single thread dump is not very useful.  A thread may not be HUNG, but rather long running for example.  So getting a series of thread dumps spread out to compare would allow you to see if the thread stack is changing over time indicating not hung but slow.  In the case of your ExecuteStreamCommand processor, it is calling a custom python script and the waits for the return from that script.  Then comes the challenge is the thread dump indicates it is waiting on your python script return to figure out why your python scripts is hanging or taking a very long time all of a sudden. Not something that can be troubleshot through NiFi. Hope this helps you in your troubleshooting journey. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@SAMSAL  The managed Authorizer uses the file-access-policy-provider (generates the authorizations.xml if it does no already exist) and then a user-group-provider.  In your case that would make most sense to be the ldap-user-group-provider.  You may also want to use the Composite-configurable-user-group-provider (configure it with ldap-user-group-provider and file-user-group-provider).  Having both a file based provider and ldap provider allows sycning of ldap users and groups  form ldap automatically as well as the file provider allowing you to manually add non ldap user/client identities for authorization as well.    Non ldap client/user identities might be certifcate based clients like other NiFi nodes/instance, etc.. Within the file-access-policy-provider you define the initial admin identity. That user identity could be set to your ldap user account identity.  Then on first start up with managed provider, it generates the authorizations.xml file seeded with the policies necessary for that initial admin user identity to act as admin.  So you could skip the single-user-provider step. Matt ... View more

@hegdemahendra  You also need to be careful with some rest-api request as they may generate reports that are held in NiFi heap until a rest-api call is made to remove them.  Doing similar actions directly from the UI handles the multiple calls needed automatically.  So calls like these can slowly eat away at the NiFi heap impacting performance until a restart.  Provenance queries would be an example of this. Unrelated note:  NiFi Variable registry has been deprecated in favor of NiFi Parameters now.  The Variable registry functionality has been officially removed starting with the latest NiFi 2.0.0-M1 release.   If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Rohit1997jio  You could use the RetryFlowFile processor for this use case.  You will feed the "failure" relationship via a connection to the RetryFlowFile processor.  The RetryFlowfile processor will continue to route the FlowFile back to PublishKafka using the "retry" relationship until maximum number of retries configured has been exceeded.  After max retries has been reached the FlowFile would instead route to the "retries_exceeded" relationship which you can connect to a LogMessage processor.  The logMessage processor would then auto-terminate the "success" relationship. The challenge you have here is  your requirement to retry once per hour for 24 hours.  You could set the penalty duration in the PublishKafka to 1 hour.  This means that FlowFile routes to the "failure" relationship would get penalized for 60 mins.  The RetryFlowFile would not consume that FlowFile from input connection until penalty duration ended.  Then configure your number of retries in the RetryFlowFile processor to 24.  Be careful with setting queue size to 250 on the failure connection.  If you reach 250 queued on the failure relationship, it will trigger backpressure on the PublishKafka processor meaning the publishKafka processor would not get scheduled again until that backpressure is gone. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Upgrading NiFi to latest version NiFi-1.23.2 helped me on this issue and there are no issues with Base64EncodeContent processor after configuring repository encryption @MattWho Thanks you! ... View more

@SowmyaP, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  ... View more

@MattWho Thanks for the response. We are using Nifi 1.21.0 version and we are using it internally in our application. Hence, we are currently planning to have Single-User-Authorizer as of now. In future, we will go for authentication provider.  Is there any way to change that 8hours expiration time? And what is the time of expiration I get from the api - /access/token/expiration?. The token is not getting expired with the time we get from that api. ... View more

Thanks, Matt! We've been pressing hard for a year now to get some migration work done from an outmoded ETL tool, and as I've moved along, there's a lot I haven't stopped to truly understand. I had seen the notice about variable registry only before, but didn't truly appreciate what that meant. Now I do! And btw, I solved the problem by calling the "udpate" API directly from an InvokeHTTP processor where there's no restriction on using attributes. Works like a charm! ... View more