Member since
07-30-2019
3391
Posts
1618
Kudos Received
1000
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 281 | 11-05-2025 11:01 AM | |
| 165 | 11-05-2025 08:01 AM | |
| 502 | 10-20-2025 06:29 AM | |
| 642 | 10-10-2025 08:03 AM | |
| 405 | 10-08-2025 10:52 AM |
12-26-2023
11:12 AM
@alexduta Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
12-26-2023
08:41 AM
Hi @MattWho thank you for your prompt response. I will try to add this onto Apache NIFI Jira as suggested. Thanks Ravi
... View more
12-22-2023
09:52 AM
@Rohit1997jio, I was going to answer this similar to @MattWho who beat me to it but I will post my answer anyway in case it can help. Basically @MattWho is correct , you can use the ExecuteScript processor to somehow simulate the retry flowfile processor and its better than RouteOnAttribute Option because when you use penalize the processor is setting idle during this time unlike the RouteOnAttribute where its always looping to unmached relationship for the period of wait time. Anyway here is my solution and in my case you dont need RouteOnAttribute but you have to add more code. I'm using groovy code for my script. The process relies on two attributes: 1- totalRetry: which incremental value to track the retry threshold every time the file is sent to retry. first time it will be set to 0. 2- isPenalized: is used to track if the file should be penalized before each retry (isPenalized == null) or its already penalized which means its ready for the next retry. The groovy script: flowFile = session.get()
if(!flowFile) return
// get totalPenalized and isPenalized attributes
totalRetry = flowFile.getAttribute('totalRetry')
isPenalized = flowFile.getAttribute('isPenalized')
// if its the first time set the value to 0 (no rety yet. first penalize)
totalRetry = !totalRetry ? 0 : totalRetry.toInteger()
// if the total retry has passed the threshold ( 3 in this case) then send to failure rel (expired).
// Total wait time (penalize time) 3*10 sec = 30 secs
if(totalRetry.toInteger()>3)
{
session.transfer(flowFile, REL_FAILURE)
return
}
// if totalRetry has not passed the threshold and the file is not
// penalized (isPenalized == null) then penalize and send back to upstream queue
if(!isPenalized)
{
flowFile = session.putAttribute(flowFile, 'isPenalized', '1')
flowFile = session.penalize(flowFile)
session.transfer(flowFile)
return
}
// Otherwise file has been already penalized then send to retry and increment totalRetry
flowFile = session.putAttribute(flowFile, 'totalRetry', (totalRetry+1).toString())
flowFile = session.removeAttribute(flowFile, 'isPenalized')
session.transfer(flowFile, REL_SUCCESS)
return You can set the Penalize period under processor SETTINGS tab Hope this helps Thanks
... View more
12-21-2023
07:11 AM
1 Kudo
@Rohit1997jio Most NiFi processors that are "started/running" are not continuously consuming threads. The processor configurable scheduling controls how often the component processor requests a thread from the NiFi controller thread pool to execute it's code. By default processors have a run schedule of 0 which means that the processor would get scheduled to run as often as possible. Should execution result in no processing of data (FlowFiles), NiFi controller will delay next scheduled execution of that processor for 1 sec before it can be scheduled again (this logic prevents excessive CPU usage by processor components). The Kafka processors simply execute Kafka Client libraries and support adding dynamic properties to the processors configuration that impact that client library behavior. The Apache NiFi documentation for each component will identify if "Dynamic Properties" are supported: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-2-6-nar/1.24.0/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6/index.html In the case of the consume and publish Kafka component processors, dynamic properties are supported. The dynamic properties are limited to those allowed by the specific kafka client library version: https://kafka.apache.org/documentation.html#configuration If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-11-2023
04:07 PM
@edtech Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
12-11-2023
07:22 AM
@Knowledgeknow You can't enable authentication and authorization on an unsecured NiFi (HTTP). To enable security in NiFi, step one is to configure HTTPS (This will require you to have certificates for all your NiFi nodes). The following configuration files have configurations related to securing your NiFi. nifi.properties --> (framework configuration file has bits related to authentication and authorization). You would enable security on your NiFi by configuring HTTPS. Once NiFi is configured with an HTTPS port authentication via TLS certificates is enabled (Can NOT be disabled and is always first method attempted to authenticate a user/client). Teh following section of this file pertain to security: Security Properties Identity Mapping Properties OpenID Connect - Since you mention Oauth2 and others... Login-identity-providers.xml --> (authentication related) used if you want to enable user authentication support through ldap or kerberos. To enable the ldap-provider or kerberos -provider, you'll need to specify one or the other in the nifi.properties configuration property: "nifi.security.user.login.identity.provider". Out-of the-box NiFi has this configured to use the Single-User-Provider (not intended for production use). Lightweight Directory Access Protocol (LDAP) - Since you mention LDAP base authentication. Once you have decided on your authentication method of choice, you'll need to setup Multi-Tenant Authorization. Authorization is used to control what your various successfully authenticated users/client have access to within NiFi's UI. This gets configured in the authorizers.xml (order in which you add various providers to this configuration file is very important!!!). This file consists of only one Authorizer (out of the box it uses the single-user-authorizer. The "authorizer" is always at the very bottom of the authorizers.xml.file. Below is a very common example structure (top to bottom order of providers added to file: FileUserGroupProvider LdapUserGroupProvider Composite Implementations - You'll want use "CompositeConfigurableUserGroupProvider". This is then configured to use both above UserGroupProviders. FileAccessPolicyProvider - configured to use above "composite-user-group-provider". StandardManagedAuthorizer - configured to use above "file-access-policy-provider" Example configuration of above: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups The authorizers.xml will setup the initial required properties for the user/client you define in the fileAccessPolicyProvider "Initial Admin Identity" (the user identity configured in this provider must be returned by ONLY one of the configured UserGroupProviders. So do NOT configure the initial admin identity in the FileUserGroupProvider if that identity is going to be returned by the LDAPUserGroupProvider. Don't worry if you mess up here initially, just delete the users.xml (FileUserGroupProvider generated) and authorizations.xml (FileAccessPolicyProvider generated) files and on next startup they will be created again. Once you have a working authentication and authorization setup, you will be able to define authorizations, using your InItial Admin user, for your other synced directly through the NiFi UI. You can also define additional authorization for your admin user (is not given access to everything, but is given admin authorization which means this user can set new authorizations for all user including itself. If you run it to authorization issue after setup, you'll want to inspect the nifi-user.log. This log will show the exact case sensitive user/client identity. If it does not match exactly with the identity that was returned by the authorizer UserGroupProviders, you'll need to go back and make some configuration changes until they do. Have fun in your journey.... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-11-2023
07:22 AM
@Knowledgeknow You can't enable authentication and authorization on an unsecured NiFi (HTTP). To enable security in NiFi, step one is to configure HTTPS (This will require you to have certificates for all your NiFi nodes). The following configuration files have configurations related to securing your NiFi. nifi.properties --> (framework configuration file has bits related to authentication and authorization). You would enable security on your NiFi by configuring HTTPS. Once NiFi is configured with an HTTPS port authentication via TLS certificates is enabled (Can NOT be disabled and is always first method attempted to authenticate a user/client). Teh following section of this file pertain to security: Security Properties Identity Mapping Properties OpenID Connect - Since you mention Oauth2 and others... Login-identity-providers.xml --> (authentication related) used if you want to enable user authentication support through ldap or kerberos. To enable the ldap-provider or kerberos -provider, you'll need to specify one or the other in the nifi.properties configuration property: "nifi.security.user.login.identity.provider". Out-of the-box NiFi has this configured to use the Single-User-Provider (not intended for production use). Lightweight Directory Access Protocol (LDAP) - Since you mention LDAP base authentication. Once you have decided on your authentication method of choice, you'll need to setup Multi-Tenant Authorization. Authorization is used to control what your various successfully authenticated users/client have access to within NiFi's UI. This gets configured in the authorizers.xml (order in which you add various providers to this configuration file is very important!!!). This file consists of only one Authorizer (out of the box it uses the single-user-authorizer. The "authorizer" is always at the very bottom of the authorizers.xml.file. Below is a very common example structure (top to bottom order of providers added to file: FileUserGroupProvider LdapUserGroupProvider Composite Implementations - You'll want use "CompositeConfigurableUserGroupProvider". This is then configured to use both above UserGroupProviders. FileAccessPolicyProvider - configured to use above "composite-user-group-provider". StandardManagedAuthorizer - configured to use above "file-access-policy-provider" Example configuration of above: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups The authorizers.xml will setup the initial required properties for the user/client you define in the fileAccessPolicyProvider "Initial Admin Identity" (the user identity configured in this provider must be returned by ONLY one of the configured UserGroupProviders. So do NOT configure the initial admin identity in the FileUserGroupProvider if that identity is going to be returned by the LDAPUserGroupProvider. Don't worry if you mess up here initially, just delete the users.xml (FileUserGroupProvider generated) and authorizations.xml (FileAccessPolicyProvider generated) files and on next startup they will be created again. Once you have a working authentication and authorization setup, you will be able to define authorizations, using your InItial Admin user, for your other synced directly through the NiFi UI. You can also define additional authorization for your admin user (is not given access to everything, but is given admin authorization which means this user can set new authorizations for all user including itself. If you run it to authorization issue after setup, you'll want to inspect the nifi-user.log. This log will show the exact case sensitive user/client identity. If it does not match exactly with the identity that was returned by the authorizer UserGroupProviders, you'll need to go back and make some configuration changes until they do. Have fun in your journey.... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-08-2023
12:05 PM
@Zifo1
Have you been able to resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
12-07-2023
12:27 AM
@Coordinador, have any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
12-06-2023
07:03 AM
Hi @Fayza , You need to setup the Content-Type as required by the API. There should be "Request Content-Type" property where you can set the value. Also any custom header values can be added as Dynamic Property. The invokehttp processor should be very flexible to accommodate the different API requirements and request types.
... View more