@AlokKumar  You'll want to use the invokeHTTP processor which can be configured to utilize an Oauth2 Access token provider: JWTBearerOAuth2AccessTokenProvider StandardOauth2AccessTokenProvider Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@oka  Perhaps others in the community may hav additional suggestions here, but since the "-" is not a valid character in JMS, you would need to use a AMQP processor to support these headers. As mentioned before there is a https://issues.apache.org/jira/browse/NIFI-14670 jira for adding AMQP 1.0 support to ConsumeAMQP processor, but it is still open and unassigned. Now that jira points to using the Qpid JMS Client in ConsumeJMS and as you experienced it works but still has limitations.  Those limitations impact these specific properties with the "-" in the name.  I would suggest adding your experience with trying to use Qpid AMQP in the above jira and what impacts it has on the two headers you require to maybe push the Apache community to adding AMQP 1.0 support to the AMQP specific processors. Additionally, there is this jira (https://issues.apache.org/jira/browse/QPID-4992), where an individual expressed some success preserving the content type header by using ActiveMQ JMS API instead of the Qpid AMQP JMS API.  So you may want to give this jira a read and maybe try this for yourself.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@fnimi  Yes a write lock is still created  when you use version control. So you'll still deal with read request stacking up until all jetty threads are full resulting in the 503.  The large commit has to be completed successfully before status within NiFi Ui can reflect the current version control status on the Process group.  Version controlled process groups also remove parses the json to remove all sensitive properties.  Version controlled dataflows can also contain parameter contexts if used and controller services they utilize making them even larger. Having 1000+ components in a single version-controlled Process Group is considered an anti-pattern in NiFi. It makes version control, deployments, and UI responsiveness incredibly slow.  Version controlled flow are meant for easy reuse or re-deployment to other NiFi clusters.  Version controlling such large flows reduce the reusability of them.  Modularize your Flows (Nested Versioning) Instead of versioning the top-level Process Group that contains everything, break your flow into smaller, logical, nested Process Groups (e.g., 50–100 components each).  Avoid versioning a PG  within a PG that is already version controlled. You can version control these smaller Process Groups individually. This drastically reduces the serialization time, lock duration, and payload size sent to the Registry. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Former Member  Do you observe same issue if you run it more frequently so it is constantly checking for new files to list rather then larger batches every 30 minutes? Matt   ... View more

@fnimi  When you say "make a new version of", are you referring to committing a new version of a version controlled process group to the NiFi-Registry service or creating a new NiFi template?  NiFi templates consume a lot of heap memory which is the leading reason why they were deprecated in Apache NiFi 1.x and fully removed in Apache NiFI 2.x major releases. Do each of these copied 4-5 big process groups contain 300 - 1000 processors or 300 - 1000 total.   When you copy components on the canvas you are creating a flow snippet that contains all the components, configurations, connections, etc of what you have selected.  That snippet is held in heap memory and when you paste it, it creates a request that contains that large snippet that must be replicated to all the nodes in a NiFi cluster.  Also when you paste NiFi must iterate through all the components in that snippet to calculate and set new x,y coordinates, component UUIDs, remap connections because of UUID changes and instantiate those components on the canvas. Also keep in mind that even after all components are added from the paste operation they are then passed along to the validator to check if each has a valid configuration and referenced services are good state and enabled. This operation creates an exclusive write lock until the above completes.  Meanwhile, your browser (and the browsers of any other users that may be open to this NiFi) continues to fire status update requests every so many seconds. These read requests are blocked waiting for the Write Lock to release.  The Jetty service has a thread pool where these read request start stacking up until no additional can be accepted and thus 503 can be encountered.  Also things like CPU thread concurrency, disk I/O, etc utilized during this large request ma be under contention. Instantiating flows from NiFi-Registry or via flow definitions imports is going to use more efficient methods then an in browser copy and paste.  I would avoid copy and paste of large flows and focus on copy paste of smaller snippets at a time.   Also want to note that Apache NiFi 1.19 is more then 3.5 years old and to keep up with critical CVEs and bug fixes you just plan regular upgrades to newer versions. The Apache NiFi 1.x major release branch is end of life and no longer receiving any new updates, fixes, or  security CVE changes in Apache.  The new Apache NIFi 2 major release branch is what is being supported in Apache now. Hope this helps explain your observations: Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  Why are you posting the same response over and over again every few hours? It makes the thread unnecessarily noisy. Your issue is a timeout issue within the SMBJClientProviderService that is attempting to establish a connection with your target SMB share.  This is taking longer then the configured timeout.  The subsequent request is probably working because the connection is still established at time of second run.  Same issue with the FetchSMB it appears as the connection is closed at time of attempting to fetch the File content.  Have you tried increasing the Timeout setting in that controllers service from the default 5 secs to 20 seconds? Have you inspected your network latency between NiFi host and SMB server?   Are you seeing any packet loss? Have you inspected the SMB server logs in Windows? Does the behavior change if you stop using the "run once" option and just start the processor?  Your run schedule of every 30 seconds is allowing the connection to timeout.  If you also adjust that run schedule to 10 seconds on ListSMB, do you see a change in behavior? Unfortunately I do not have a SMB share available to test this setup myself and am providing what guidance and suggestions I can to help with your community query. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@oka  I believe your issue is that "-" is not a valid character in the JMS specification. https://stackoverflow.com/questions/30022453/how-to-set-content-type-header-for-jms-message While the "-" is valid for AMQP messages. There is a ConsumeAMQP processor but I see it only supports AMQP version 0.91 You don't have a issue connecting but just sound like these specific headers are ignored since they contain that invalid JMS character. There is an existing Apache NiFi jira (https://issues.apache.org/jira/browse/NIFI-14670) for adding AMQP 1.0 support to the AMQP processors; however, it is open and unassigned. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fnimi   I am not clear on "It dies for 15-30 minutes".    What version of Apache NiFi are you using? Is this a standalone instance of NiFi or a multi-node NiFi Cluster? Does the NiFi child process die and get launched agin by the NiFi bootstrap process?  Any exceptions thrown in the nifi-app.log or nifi-bootstrap.log (timeouts, OutOfMemory(OOM), etc)?  Are you seeing a long stop-the-world Java Garbage Collection event? How large are these "big" process groups? The more detail you can provide, the better chance a community member will be able to provide some suggestions. Thank you, Matt ... View more

@nisaar  I see you set retry on the ListSMB processor "success" relationship.  You should never set retry on any "Success" relationship.  The connection retry logic is triggered when a FlowFile would be routed to the relationship.  Instead of routing the FlowFile to the relationship it remains on the upstream connection to be "retried".  That means that only when FetchSMB successfully lists a file will it trigger retry.  Plus retry implies a there is some upstream location to persist a source FlowFile. Only once all retry attempts are exhausted will the FlowFile actually be placed on the connection containing the "success" relationship. I would expect unexpected behavior with such a configuration.  Using "retry" on a failure or other error type relationship makes sense since these are the relationships you don't expect FlowFiles to be routed to.  so makes sense on your FetchSMB processor so that source FlowFile will remain on the inbound connection to FetchSMB until both the configured retry attempts are made.  I both retries fail I see you terminate the "Failure" relationship which effectively throws away that file.   I don't know that terminate is what i would do here either since ListSMB was successful in order to produce FlowFile waiting at FetchSMB.  Plus state is recorded by ListSMB and the file that failed ot be fetched would never get listed again unless you touched file to update last modified time or cleared state on listSMB processor.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more