@asand3r  JVM Garbage collection is stop-the-world which would prevent for the duration of that GC event  the Kafka clients from communicating with Kafka. If that duration of pause is long enough I could cause Kafka to do a rebalance.  I can't say that you are experiencing that .   Maybe put the consumeKafka processor class in INFO level logging and monitor the nifi-app.log for any indication of rebalance happening. When it comes GC pauses, a common mistake I see is individuals setting the JVM heap settings in NiFi way to high simply because the server on which they have in stalled NiFi has a lot of installed memory.   Since GC only happens once the allocated JVM memory utilization reaches around 80%, large heaps could lead to long stop-the-world if there is a lot top clean-up to do.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thank you for replying, that's the exact solution I eventually settled on.  Best, Shelly ... View more

@Alexm__  While i have never done anything myself with Azure DevOps pipelines, I don't see why this would not be possible.  Dev, test, prod environments would likely have slight variations in NiFi configurations (source and target service URLs, passwords/usernames, etc).  So when designing your Process Group dataflows you'll want to take that into account and utilize NiFi's Parameter contexts to define such variable value configuration properties.  Sensitive properties (passwords) are never passed to NiFi-Registry.  So any version controlled PG imported to another NiFi will not have the passwords set. Once you version control that PG, you can deploy it through rest-api calls to other NiFi deployments.  First time it is deployed it will simply import the parameter context used in source (dev) environment.  You would need to modify that parameter context in test, and prod environments to set passwords and alter any other parameters as needed by each unique env.   Once the modified parameter context of same name exists in the other environments, promoting new versions of dataflows that use that parameter context becomes very easy.  The updated dataflows will continue to use the local env parameter context values rather then those used in dev.  If a new parameter is introduced to the parameter context, is simply gets added to the existing parameter context of the same name in test and prod envs.   So there will be some consideration in your automated promotion of version controlled dataflows between environments to consider. Versioning a DataFlow Parameters in Versioned Flows Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@yoonli  It would be helpful if you shared the complete authorization exception you are encountering.  I have a feeling your authorization exception is not related to your server certificate, but more related to your individual NiFi user. Using a load balancer in front of your NiFi cluster would require that session affinity (sticky sessions) is enabled in your load balancer.  The why? Any login based user authentication (ldap-provider, kerberos-provdier, etc) result in a token being issued to the user and a server side token stored on the NiFi server that issues the client token. Only the specific node in the NiFi cluster that issued the client bearer token will have the corresponding server side token. If your load balancer does not have sticky sessions enabled subsequent requests after obtaining the client bearer token may get direct to a different node in the cluster. Your browser will include this client token in all subsequent request to NiFi Since the other nodes will not hav the corresponding server token for your user the session would result in an not authorized response.  Possible helpful HAProxy links: https://www.haproxy.com/blog/enable-sticky-sessions-in-haproxy https://www.haproxy.com/solutions/load-balancing ---- Certificate based authentication is not an issue since the client/server MutualTLS exchange happens in every communication between client and server.   This is why is suspect that your setup involves a login based authentication method. ---- I see you configured your LB IP in the nifi.web.proxy.host property within the nifi.properties file.   This property has nothing directly related to client/user authentication.  It is about making sure NiFi accepts requests destined for a different hostname/IP then the destination host that received it. Let's say you initiate a connection to URL containing host: https://10.29.144.56/nifi/  Your HAProxy then routes that request to NiFi on host 10.29.144.58 which returns a server certificate with that servers hostname or the IP 10.29.144.58.  The connection is going to be blocked because it appears as a man-in-the-middle attack.  The expectation was that the request would be processed by the server 10.29.144.56; however, host 10.29.144.58 received the request.  By adding 10.29.144.56 to the proxy.host property in NiFi, you are telling NiFi to accept requests intended for a different hostname or IP then the actual NiFi's hostname or IP.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@yoonli  This thread is growing in to multiple queries that are not directly related.  Please start a new community question so the information is easier for our community members to follow when they have similar issues. Thank you, Matt ... View more

Thank you @MattWho,  I’m currently using nifi-atlassian-nar-2.5.0-SNAPSHOT.nar, even though my NiFi version is 2.4.0. ... View more

@PriyankaMondal  There significant differences between the Apache NiFi 1.x and Apache NiFi 2.x major release.  Deprecated and removed processors, controller services, and reporting task components Some components moved to new nars Deprecated and removed NiFi Templates Deprecated and removed NiFi Variable registry. This means that you can not simply move your flow.json.gz from Apache NiFi 1.23.2 to Apache NiFi 2.x.  First you should update your dataflows so they are no longer using any of the deprecated components in Apache NiFi 1.x.  I recommend first upgrading to Apache NiFi 1.28 so you have the latest deprecation logging.  Apache NiFi 1.28 should produce a deprecation log that will tell you all the deprecated components you are currently using on your dataflow canvas.  Take steps to remove these components or replace them with alternative components that are still available in Apache NiFi 2.x. Examples:  JoltTransformJson processor was included in the  nifi-standard-nar in Apache NiFi 1.x , but has moved to a nifi-jolt-nar on Apache NiFi  2.x.  So the class name has changed.  You'll need to add the Apache NiFi 2.x class version of JoltTransformJson processor to canvas and reconfigure  it for your dataflow and delete the ghosted (dashed line around it because NiFi does not know that class)  JoltTransformJson processor.  While you manually changed the version in yoru flow.json.gz, you did not change the class path as needed resulting in the processor still being ghosted. ConvertAvroToJson processor was deprecated in NiFi 1.x and removed in Apache NiFi 2.x.  Would need to replace it with a convertRecord (available in Apache NIFi 1.x and 2.x) configured to use an Avro Reader and Json Writer. NiFi Variable Registry removed.  So if you are using any NiFi variables in your processor configuration in Apache NiFi 1.x, you'll need to modify your dataflows to use NiFi parameters instead (Parameters exist in Apache NiFi1.x and 2.x) Templates were deprecate in NiFi 1.x and were replaced with Flow Definitions. Templates removed in NiFi 2.x. Would need to remove all templates saved in NiFi before moving to Apache NiFi 2.x. Above is just a short list. refer to the deprecation log produced by Apache NiFi 1.28 to see all deprecated processor you may have been using in yoru dataflows. I do wish there was an easier way to move from Apache NiFi 1.x to 2.x, but depending on your use of deprecated features and changed component classes, there may be little too a lot of effort needed to prepare yoru NiFi 1.x datfalows for migration to NiFi 2.x For Cloudera Flow Management license holders: For Cloudera Flow Management NiFi users, Cloudera has built a Cloudera Flow Management Migration Tool that automated many of the migration steps (swapping processors when alternatives exist, changing processor classes to new classes, converting templates to flow definitions, converting NiFi variables to NiFi parameters, etc.  While there is still no direct upgrade possible from Cloudera Flow Management 2.1.7 (Apache NiFi 1.x based) to Cloudera Flow Management 4.10 (Apache NIFi 2.x based), this migration tool takes a lot of the manual work out of preparing your flow.json.gz for the new major release. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Virt_Apatt  I don't know enough about your use case to make any other suggestions. All I know is that your user(s) supply some custom date that you have NiFi add 10 days to before running a Oracle query to get some result set returned to NiFi. NiFi is typically used to build dataflows that are always in the running state, so users do not need to continuously stop, modify component(s), and start a dataflow/component. What is the significance of this "custom date" that starts your dataflow? Is there any pattern to these custom dates? Can the next custom date be derived from the response from the previous Oracle query? How often does this dataflow get executed? Just some examples (there are many NiFi processor components that can fetch content from external sources): You could start your dataflow with a getSFTP or getFile processor that is checks a specific source SFTP server or local directory for a specific filename. In that file is your custom date.  You then build your dataflow to extract that custom date from the consumed file to then execute your oracle query.  This way your NiFi is always running and just waiting for the next file to show up on the SFTP server or in that local directory it keeps checking. Or maybe setup an http lister (ListenHTTP or HandleHTTPRequest) that listens for an http post that contains the custom date needed for your running dataflow.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more