@AlokKumar   You are correct that the ConsumePOP3 processor does not support an inbound connection to it.    Even if it did, the username and password fields do not support NiFi Expression language allowing you to pass either of those values in from the source FlowFile.   There aren't any other native processors that can support this dynamic credentials use case.   You would need to create a custom script that could be called by scripting processors or create your own custom processor.  ExecuteScript ExecuteProcess ExecuteGroovyScript The reason processors like ConsumePOP3 do not support inbound connections is because they are designed to execute continuously on a run schedule and produce an individual FlowFile for each new email message consumed.   So supporting an inbound connection raises the question... What do you do with the source FlowFile that you would use as the trigger?  Then you also have the challenge of continuously consumption.  You would need to keep producing an input FlowFile for each email account to make sure you keep consuming from each source account.   Plus this processor does not write any attributes to outbound FlowFile to distinguish which account message came from.    Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  The ListSMB processor only fetches metadata about the files in the target SMB location.  For each file found it creates a 0 byte NiFi FlowFile that includes a bunch of metadata that can be used to fetch the content later by the FetchSMB processor.   The List<type> and Fetch<type> processors are used to make sure one node in a multi-node NiFi cluster si not doing all the heavy work.  The List<type> processor would be configured to run on "Primary Node" only.  The success relationship would be connected to the FetchSMB via a connection.  That connection would the need to be configured to load balance the 0 Byte FlowFiles across all your NiFi nodes so that each could Fetch a fair share of the content and process a fair share of the workload of this dataflow.  What are the difference between the files that fail on content fetch versus those that are successful?  Are these files larger resulting in a timeout exception? Are those that are timing out always being fetched by one specific node in your NiFi cluster?   Have you verified the all nodes can successfully connect to the SMB server? Have you tried increasing the timeout set in the SmbjClientProviderService used by the SMB processors?  Try setting it to 60 seconds or higher to see if the failed files can successfully fetch the content from SMB.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Vishesh  Sorry, but there is not enough information to give a definitive cause for your issue.  There is typically full stack traces to go with these types of logged exceptions which can tell us more, but even then it may not be enough information still.   As i mentioned before there are numerous improvements between Apache NIFi 1.23 and the last Apache NIFi 1.x release version 1.28.  I'd strongly encourage you to upgrade to see if your issue persists.   You may be hitting this bug  NIFI-12232  which was addressed in versions 1.26+.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Vishesh  It is not possible to say exactly what issue you may have encountered here.  Do you still have the complete stack trace that followed the node disconnection exception?  It is likely to have some "Caused by:.." lines in the full stack trace that may help. Any changes being done when the disconnection occurred?   Stopping/starting a dataflow or process group? Adding a template to the canvas?  (if you are still using templates, they were deprecated in NiFi 1.x and Flow definitions are the new method to use.  Templates were officially removed in NiFi 2.x).  I have corruption issues related to templates previously. When you restarted your service, what observations were made in the nifi-app.log on all three nodes during startup?  A flow election happens first where like flows each get a vote, the flow with the most ote becomes the cluster flow and nodes without that flow will join and inherit that cluster flow. One of your three nodes would have been elected as cluster coordinator and all other nodes would have formed the cluster by sending heartbeats to that node. During that node connection phase, any node with a mismatched flow would inherit the cluster flow.  Any logging related to one or more of your nodes inheriting the cluster flow on startup? If not that then could be possible that some component was stuck in an enabling component state.  So when you start a component(s) on the canvas, the component goes initially to "starting" and then "started".  Likewise, stopping a component transitions to "stopping" and then "stopped".  You may have been in situation where your nodes had a component stuck in the "stopping" or "starting" phase, but your cluster coordinator completed the transition.   This could be caused by a bug in the component, load on the system, component has very long running process or hung process working on a FlowFile with large content, etc...   Inspecting some thread dumps from those disconnected running nodes might help identifying scenario.   This might be the most likely cause for you?  I say this because if your flow.json.gz was corrupt, restarting your cluster would have had exceptions when trying to load the corrupted flow.json.gz.  When stopping the nodes, NiFi eventually times out waiting for nodes to gracefully completed running threads and kills them.  Then on restart no flow.json.gz corruption, all nodes restart fine, flow loaded successfully, and set the components to same running state. While none of above is a definitive answer because there is not enough info to provide that, hopefully this give you an idea of what could have happened so you know what to collect or look at deeper should it happen again.   I will add that a number of fixes have gone into the newer releases (some around NiFi clustering).  Apache NIFi 1.x is officially end of life.  If you can not migrate to the newer Apache NiFi 2.x branch, you should at least upgrade to the latest Apache NiFi 1.28 release to take advantage of fixes done there since 1.23. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@oka  Can you share more details about your Apache NiFi version and dataflow configuration? Full Apache NiFi version PublishJMS processor configuration Sample FlowFile metadata/attributes of a FlowFile queued to the PublishJMS processor (exclude any sensitive values). Which FlowFile contains the attribute string you expect to be sent to JMS endpoint These details can be helpful in evaluating your setup and possible issue/solution. Thank you, Matt ... View more

@Frank168  I'd expect what you are seeing with the long sync times.  Sync interval has not impact in speed of the user an group syncs.  This simply controls how often NiFi is expected re-sync with ldap.  So in your case, NiFi would finding one sync and immediately start the next.  NiFi does a sync during startup as it loads the authorizers.xml, NiFi does not continue to load until that initial sync completes otherwise it can't do authorizations.   After that initial successful sync with ldap, NiFi will finish loading and only then will UI become available.   While running the sync interval will run in the background to re-sync from ldap in case any users or groups are added/removed/updated since last sync.    some users adjust this to sync every 24 hours instead of every 30 minutes.   This depends on how quickly you want NiFi to be aware of changes. As I commented before, you are loading 50,000+ users into NiFi heap memory (this impact the free heap available to the NiFi process for executing your dataflows on the canvas) and then redoing that every sync interval.  This was not how NiFi ldap-user-group-provider was intended to be used.   The only users and groups that should be loaded in to NiFi are those you plan to establish authorization policies for and will be accessing your secured NiFi. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Krish98  Unfortunately, I have no "Best Practice" recommendation for the use case you have shared.  It is not a use case I have ever setup before. Also want to share that in Apache NiFi 2.4+ version a new JWTBearerOAuth2AccessTokenProvider controller service was introduced.  While not a solution to you query, I wanted to share this with you.  Apache NiFi jira: NIFI-14380 NOTE: The Apache NiFi 1.x major release line is End-Of-Life now.  There will be no future releases of the 1. major release line.  There is no direct upgrade path from Apache NiFi 1.x to Apache NiFi 2.x.  You'll need to migrate your dataflows from 1.x to 2.x. For our Cloudera Flow Management licensed users, we provide tooling to assist with migrating dataflows from Flow Management versions based on Apache NiFi 1.x to Flow Management versions based on Apache NiFi 2.x.   Cloudera Flow Management 2.x also includes many of the components deprecated and no longer included in the Apache NiFi 2.x release line.  Thank you, Matt ... View more

@Frank168  When NiFi fails to start, you'll see logging in the nifi-app.log with details as to why.  My guess would be it failed when loading the authorizer because the ldap-user-group-provider execution was unsuccessful.  That in turn was likely caused because you set the client page size to a value higher then the max enforced by the ldap server.  LDAP server common MaxPageSize settings are 500 or 1000.   Setting Page Size in NiFi to 500 is a common practice.  The NiFi ldap client will request all returns in pages of 500.  The Ldap server will return as many pages as required to fulfill the request.  When page size is left blank in NiFi, the ldap server is only going to return one page limiting the result set to whatever fit in the ldap server page page size which would explain your missing returns. Asa reminder, I caution against returning so  many users.    Do you expect all 1700+ users to be accessing your NiFi?   That would be extremely uncommon.   Typical usage is that ldap users that will be accessing NiFi are added to a few specific ldap groups. Then the ldap-user-group-provider is configured to fetch only those groups and group members instead of syncing everyone from ldap.  This strategy limits the heap consumed by the return user and group set.  Keep in mind that the ldap-user-group-provider default is to re-sync every 30 minutes also.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Frank168  What you expect is exactly how it works.  Did you configure any user search based filtering in your LdapuserGroupProvider?  Is your Referral Strategy set to FOLLOW?  Any exceptions in the nifi-user.log or nifi-app.log?  How many users are being returned successfully (count)? Did you set your Page Size? Ldap often limits the number of returns in a single page.  NiFi by default does not set a "Page Size" so it request all results in one page.  So depending on how many users you are trying to return and your ldap's defaults, this could result in missing users.  Try setting the "Page Size" in NiFi to 500 to see if that resolves your issue. NOTE:  All user and group identities returned by the ldap-user-group-provider are held in NiFi's heap memory.   You should avoid syncing very large sets of users and groups and utilize user and group search filtering to only return those users and groups that will be accessing NiFi. If page size is not your issue, inspecting your setup may provide other clues.   Can you share your login-identity-providers.xml and authorizers.xml file configurations (redact sensitive values)? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more