@saquibsk  Additional settings? With a Secured NiFi (which you should always be using) there is authentication and authorization involved with any rest-api request.  The simplest approach is to generate a clientAuth certificate that is trusted via the truststore your secured NiFi is configured to use in nifi.properties file.   Then that certificate is added to a keystore.  The invokeHTTP processor can be configured to use a StandardRestrictedSSLContextService that you configure with the keystore you created and the truststore that NiFi already uses that can trust that certificate. On NiFi side, you would need to add that client as a user entity so you can assign authorization policies to.  You can then authorize that client/user identity to the policies needed to start and stop specific processor components.  That policy would be the "operate the component" policy that you can set just on the QueryDatabaseTable processor or any other specific processor you want to automate. component-level-access-policies Yes, there are some initial steps to setup the keystore and truststores needed, but then those can be used over and over for all automation within NiFi you want to achieve. NiFi processors execute based on the individual processor's configured scheduling. There is no other option to stop or start individual processors except manually through the UI by an authorized user of via the rest api. NiFi was designed with an always running type architecture in mind.  Stepping our of that architecture would require extra steps or redesigning your dataflows to operate within that architecture style. If your executions always happen at set times you could use cron scheduling, but that is not going to be an optimal design for performance. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@2ir  Heap usage continues to grow until utilization exceeds ~80% at which time heap Garbage Collection (GC) is executed to release unused heap memory allocations.   Even if there are no FlowFiles or running flows in NiFi, if the GC threshold has not been reached, the heap would not be cleaned out.   First question is why is our NiFi heap set so high already?  Where you encountering OutOfMemory (OOM) issues with smaller heap settings.  You'll get the best performance using the smallest workable heap values.  Typical NiFi heap size recommendations in most scenarios is XMS and XMX of 16 GB or less. You stated that consumption was stable for 3 weeks and then changed: What changed about the content being ingested and created (size, number, etc)? Where any new dataflows added or recently enabled? What processor types are being used in your dataflows? Are you leaving any FlowFiles queued up in dataflow connections? Are you writing large attributes to FlowFiles? Are you extracting content from FlowFiles into attributes? Aside from that you would need to look at hep dumps to see what is consuming the majority of the space to maybe identify the components responsible. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@saquibsk  A couple thoughts come to mind here... Have you looked at maybe using a GenerateTableFetch processor in "Dim 2" which can be triggered by an incoming FlowFile.  This processor will take an optional inbound connection as a trigger. Other option might be to use an invokeHTTP processor after your PutDatabaseRecord processor to start the the "Dim 2" QueryDatabaseTable processor via the NiFi rest-api (REST API).  You could then do similar after DiM 2 QueryDatabaseTable processor to stop the processor again. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@2ir  NiFi can consume heap (native) and non-heap (non native) memory. This commonly happens with processor that create jetty servers for listening for inbound requests, scripting processors that execute child scripts, processors that execute OS commands,  NiFi bootstrap parent process,  etc...  So XMX heap memory is only part of the memory that can be used by a Java application.  From within the NiFi UI --> global menu --> cluster or summary, you can see that actual amount of heap utilized using (JVM tab cluster UI or summary system diagnostics UI). I would advise against setting your Heap so high when you have 47 GB total memory.  It is likely that your OS if linux based is going to invoke OOM killer to kill the NiFi process to protect the OS.  I'd advise reducing your heap usage to XMS and XMX of 24 GB. Re-evaluating your dataflows for high memory use processor and making sure they are optimally configured is next steps.  The embedded documentation for the the processor components will each have a "System Resource Considerations" section that will tell you if the processor has the potential to use high memory or high cpu.  For processors with potential for high heap usage, be careful with concurrent task configuration. Default on current tasks is always 1.  Increasing it is like adding a second copy of the processor allowing multiple concurrent executions thus increasing the heap usage. (Example: ReplaceText 1.25.0) You'll want to be careful using templates (deprecated) as any templates generated and held in NiFI consume heap.  FlowFile metadata is held in heap so avoid creating FlowFile with large attributes (like extracting content to attributes). Use Record base processor whenever possible to reduce numbers of individual FlowFiles.  Use a NiFi cluster instead of standalone NiFi to spread FlowFile load across multiple NiFi instances. Monitor heap usage and and collect heap dumps to analyze what is consuming the heap. Hope this helps you with your investigative journey. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@darkcoffeelake  NiFi out-of-the-box setup generates simply keystore and truststore automatically and set the login provider to single-user-provider and authorizer to single-user-authorizer.  This out-of-the-box setup is simplifies secured access for evaluation of NiFi.  It is not a production ready setup in that it does not support multi-user authentication, granular access controls, or NiFi cluster setups. There are bunch of steps that go into securing Apache NiFi for production ready environments.  Securing NiFi not only sets up NiFi over an HTTPS connection, but also requires that user authentication and authorization is setup.   NiFi will require a keystore and truststore which youcan create yourself or use publicly available service to create them for you (example would be tinycert).  The keystore created for you NiFi must meet the following requirements for NiFi: Contains only 1 PrivateKey entry.  Does not use wildcards in the DN of PrivateKey certificate. Has both clientAuth and serverAuth Extended key Usage (EKU) Has SubjectAlternativeNames (SAN) entry(s) matching NiFi hostname and any other name that may be used to access the NiFi.   The truststore needs to contain the complete trust chain for your NiFi keystore certficate.  A certifcate might be self signed (meaning both issuer and signer are same DN), it may be signed by an intermediate CA, or rootCA.      If signed by an intermeidiate CA, your truststore would need to have the trustedCertEntry (public key) for the intermediate CA (intermediate CA is any CA where signer and issuer are different DNs) and the trusted certEntry for that signer and so until you reach the root CA in the chain (root CA will have same signer and issuer DN). Once you have your certificates, you'll need to decide how your users are going to authenticate with NiFi.  NiFi does not have a embedded provider that supports multi-user authentication.  Here is what is available to choose from: User Authentication LDAP and Kerberos are probably the most commonly used. Once you have decided how you are going to authenticate your users, you'll need to setup authorization for those users.  here are your options here: Multi-Tenant Authorization The simplest authorizers.xml setup would utilize the  StandardManagedAuthorizer, FileAccessPolicyProvider, and FileUserGroupProvider. a sample configuration can be seen here: https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/administration-guide.html#file-based-ldap-authentication If setup correctly, on first startup, the above authorizers.xml will generate and seed the users.xml and authorizations.xml file so that your initial admin user (a ldap user or kerberos user for example) with the necessary authorization policies to access the NiFi UI.  From the NiFi UI, that initial admin user can setup additional user identity authorizations. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Vas  The straight out of the box generated keystore and truststore will not have "nifi.local" as a SAN entry.   You could generate your own keystore and truststore with needed SAN entry(s). If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Chaitanya_Y  I am not sure why the Apache NiFi Community did not release and migration guidance with Apache NiFi 1.25 release.   However, there are release notes that highlight notable changes: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.25.0 You will want to read through all the migration guidance for every release between 1.16 and 1.25 to see if anything applies to your specific setup or dataflows.  Take not of any deprecated components you may be using currently and any components that were remove form the default release (removed does not mean gone, you can download those removed nars from central repository and add them to the 1.25.0. release if needed. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@broobalaji  HDF 1.8.0.3.3.1.0-10 was released way back in 2017.  I strongly recommend upgrading to a much newer release of CFM.   NiFi Templates have been deprecated and are completely removed as of Apache NiFi 2.x releases.  Apache NiFi deprecated templates for a number of reasons: 1. Templates uploaded to NiFi (even if not instantiated/imported to the NiFi canvas reside within NiFi's heap memory space)  2. Large uploaded templates or many uploaded templates can have a substantial impact on NiFi performance because of the amount of heap they can consume.  Simply increasing the size of NiFi's heap is also not the best solution to that heap usage as large heaps just lend themselves to longer stop-the-world garbage collections with the JVM.  3. Apache NiFi deprecated and moved away from using xml based flow in favor of json flow definitions around the Apache NiFi 1.16 time frame.   Flow definitions (JSON files) can exported and imported without uploading them in to heap memory within NiFi. The above info aside.... It is best to use the developer tools available in your web browser to inspect/capture the rest-api call being made when you perform the same steps directly via the NiFi UI.  This makes it easy to understand the calls that need to be made in your automation.  I also encourage you if you continue to use templates to upload, import to UI, and then delete the uploaded template to minimize heap impact. Thanks, Matt ... View more