Member since
07-30-2019
3472
Posts
1642
Kudos Received
1021
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 205 | 06-15-2026 08:08 AM | |
| 388 | 06-03-2026 06:06 PM | |
| 617 | 05-06-2026 09:16 AM | |
| 1379 | 05-04-2026 05:20 AM | |
| 710 | 05-01-2026 10:15 AM |
02-27-2025
11:14 AM
@AlokVenugopal Welcome to the community. What you are encountering is an authorization issue and not an authentication issue. NiFi is accepting your token issued through your application login, but then authorization does not exist for the user identity derived from you token. In NiFi, after successful authentication, the user identity is passed to the NiFi authorizer to determine what NiFi policies have been authorized for that user identity. When using yoru application's token, this result in no authorization found because neither the user Identity or any known groups that user identity belongs to are authorized for the required policy. identity[kLM-4Eld2dZnX_dD3iB0df2fTvXQxa1J2ffdLoK-ozas], groups[] Supporting the user "unique id" would require that NiFi's authorizer contained that unique id and it was authorized to the necessary NiFi policies. Authorizing users based in these unique id does not make much sense in NiFi as it would be error prone and difficult to manage authorization. An Admin would need to know what user these unique ID map to in order to setup authorization successfully. The first option would be modifying your app so that the returned token contain and ID that matches the user identity similar to what NiFi does. Assuming this "unique id" does not change and is always the same for the specific user, perhaps you can work around this creatively within NiFi through group based authorization. This would requiring using the file-user-group-provider within the NiFi authorizers.xml. This will allow you to manual add user identities and group identities. So you create a new group such as "username" via the NiFi UI. You then add your existing user (the one that successfully gets authorized when you authenticate through NiFi) to this new group. You then add a new user identity for that "unique id" and make that new user a member of that same group via the NiFi UI. Now authorize the group to whichever policies are necessary. Now no matter if your user authenticates via NiFi to get token or through your app to get a token, the user will successfully be authorized via the shared group membership. Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
02-26-2025
12:31 PM
Thank you @MattWho . Thanks for brief explanation. I understand the loop is causing large number of queue. Let me redesign the flow. Thanks !
... View more
02-26-2025
05:52 AM
@dsender Apache NiFi is a data agnostic service. It can move any data format through a dataflow because the content is treated as just bytes inside a FlowFile. The only time the content needs to be read is if there is need to manipulate it, extract from it, etc. Then you would need to use a processor that understand the data format. While it does not appear that Cloudera Flow Management offers any SAS specific processor components. So some custom processor would need to be developed or perhaps you can use one of the available scripting processors? You would still need to write a custom script to ingest and/or process the SAS files. So this starts with the question of how would you pull these SAS files from command line outside of using NiFi? Then figure out how to turn that success into a custom script or processor that does the same thing. You could also reach out to your Cloudera Account owner and discuss possible professional service offering that maybe able to help you here with your custom needs. Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
02-25-2025
07:47 AM
@Jaydeep Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
02-25-2025
05:49 AM
@dan_lucas From the exception, this appears to be a configuration issue most likely. You'll want to verify the NiFi Expression Language statement used in the putFTP processor's "Remote Path" property. I assume you have something configured there like ${absolute.path}/${airlinename} in that property? If you manually connect to the FTP server can you successfully navigate the path? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
02-21-2025
07:13 AM
The original poster of the thread has the ability to mark the solution @sujith18. @ravi_tadepally has @MattWho 's response helped you overcome your issue? If so, please mark his reply as the solution.
... View more
02-20-2025
06:40 PM
Hi @MattWho , Thank you for the quick response. I have changed the claim to upn, but the issue remains the same. You mentioned that I am using OpenID, which relies on the NiFi service for authentication. Is there an alternative way to authenticate directly using Azure AD? If so, what configuration changes should I make? Sorry if my question doesn't make sense. Like i said i am new to Nifi Thank you in advance for your support! I am attaching logs 2025-02-21 06:26:47,841 INFO [NiFi Web Server-117] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[kLM-4Eld2dZnX_dD3iB0df2fTvXQxa1J2ffdLoK-ozas], groups[] does not have permission to access the requested resource. Unable to view Process Group with ID fa3a74d9-0194-1000-85b2-d20263f3e54b. Returning Forbidden response.
2025-02-21 06:26:47,841 DEBUG [NiFi Web Server-117] o.a.n.w.a.c.AccessDeniedExceptionMapper
org.apache.nifi.authorization.AccessDeniedException: Unable to view Process Group with ID fa3a74d9-0194-1000-85b2-d20263f3e54b.
at org.apache.nifi.authorization.resource.Authorizable.authorize(Authorizable.java:285)
at org.apache.nifi.authorization.resource.Authorizable.authorize(Authorizable.java:298)
at org.apache.nifi.web.api.ProcessGroupResource.lambda$getProcessGroup$0(ProcessGroupResource.java:269)
at org.apache.nifi.web.StandardNiFiServiceFacade.authorizeAccess(StandardNiFiServiceFacade.java:517)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:727)
at org.apache.nifi.web.StandardNiFiServiceFacade$$SpringCGLIB$$0.authorizeAccess(<generated>)
at org.apache.nifi.web.api.ProcessGroupResource.getProcessGroup(ProcessGroupResource.java:267)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) Here kLM-4Eld2dZnX_dD3iB0df2fTvXQxa1J2ffdLoK-ozas is the user id. if i use Nifi token then no issue. application login ad works fine. But when i use the token generated from my application this error occur for same user. When i analyze the tokens, this id is there in the sub field of my token. But for Nifi token sub field is the email id. I guess i need to change the openid to direct azure AD
... View more
02-18-2025
12:49 PM
1 Kudo
@fy-test Welcome to the community. What are the reasons being given in the NiFi elected cluster coordinator nifi-app.log for the disconnection or from the disconnected node's nifi-app.log? No matter which NiFi node you are connected to any, change request must be sent the elected "Cluster Coordinator" that replicates that request to all connected nodes. If any of the nodes that has been requested to make the change fails to respond in the node will get disconnected. The elected "Primary node" is the node on which any primary only scheduled processor components will run. It is also important to understand that which node is elected as the "Primary" or "Coordinator" can change at any time. I don't think forcing all your users on to the Primary node is going to solve your issue. Even with a node disconnection caused by a failure of the request replication, the disconnected node should attempt to reconnect to the node and inherit the cluster flow if is it different from the local flow on the connecting node. You should also be looking at things like CPU load average, Heap usage, and Garbage collection stats on your primary node versus the other nodes. Perhaps adjust max timer driven thread pool sizes or adjusting timeouts would be helpful. Cluster Node Properties How well are your dataflow designs distributing the load across all nodes in your cluster? Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
02-18-2025
12:46 PM
@fy-test Welcome to the community. No matter which NiFi node you are connected to any change request must be sent the elected "Cluster Coordinator" that replicates that request to all connected nodes. If any of the nodes that has been requested to make the change fails to respond in the node will get disconnected. The elected "Primary node" is the node on which any primary only scheduled processor components will run. It is also important to understand that which node is elected as the "Primary" or "Coordinator" can change at any time. I don't think forcing all your users on to the Primary node is going to solve your issue. Even with a node disconnection caused by a failure of the request replication, the disconnected node should attempt to reconnect to the node and inherit the cluster flow if is it different from the local flow on the connecting node. You should also be looking at things like CPU load average, Heap usage, and Garbage collection stats on your primary node versus the other nodes. Perhaps adjust max timer driven thread pool sizes or adjusting timeouts would be helpful. Cluster Node Properties How well are your dataflow designs distributing the load across all nodes in your cluster? Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
02-18-2025
11:13 AM
1 Kudo
@mridul_tripathi That is not exactly the dataflow I was trying to convey, but good attempt. This is what I was envisioning: It start with fetching the of files from "SFTP1" using the listSFTP and FetchSFTP processors. The ListSFTP processor will create a bunch of FlowFile attributes on the output FlowFile that can be used by the FetchSFTP to fetch the content and add it to the FlowFile. In the FetchSFTP processor you will specify the SFTP1 Hostname, Username, and Password. You will use NiFi Expression language to tell FetchSPT to fetch the specific content based in the FlowFile attributes created by ListSFTP: Next the FlowFile (now with its content from SFTP1) is passed to the CryptographicHashContent processor that will create a new FlowFile Attribute (content_SHA-256) on the flowFile with the content hash. Unfortunately, we have no control over the FlowFile attribute name created by this processor. Next The FlowFile is passed to an UpdateAttribute processor is used to move the (content_SHA-256) FlowFile to a new FlowFile attribute and remove the content_SHA-256 attribute completely so we can calculate it again later after fetch same file from SFTP2. I created a new FlowFile Attribute (SFTP1_hash) where I copied over the hash. Clicking the "+" will allow you to add a dynamic property. Next I pass the FlowFile to ModifyBytes processor to remove the content from the FlowFile. Now it is time to fetch the content for this same Filename from SFTP2 by using another FetchSFTP processor. This FetchSFTP processor will be configured with the hostname for SFTP2, username for SFTP2, and password for SFT2. We still want to use the filename from the FlowFile to make sure we are fetching the same file contents from SFTP2. So you can still use "${path}/${filename}" assuming both SFTP1 and SFTP2 use the same path. If not, you will need to set path manually (<some SFTP2 path>/${filename}). Now you pass the FlowFile to another CryptographicHashContent processor which will have the content fetched from SFPT2 for the same filename. At this point in time your FlowFile has a bunch of FlowFile attributes (including hash of both content from SFTP1 (SFTP1_hash) and SFTP2 (content_SHA256)and only the content from SFTP2. So you'll pass it Now it is time to compare those two hash attribute values to make sure they are identical using an RouteOnAttribute processor. Here will create a NiFi Expression Language (NEL) expression to make this comparison. Clicking the "+" will allow you to add a dynamic property. Each Dynamic property added in this property becomes a new relationship on the processor. ${content_SHA-256:equals(${SFTP1_hash})} This NEL will return the value/string from FlowFile's "content_SH256" attribute and check to see if it is equal to the value/string from the FlowFile's "SFTP1_hash" attribute. If true, the FlowFile will be routed to the new "Content-Match" relationship. If false, it will be routed to the exiting "unmatched" relationship. Here you can decide if just want to auto-terminate the "Content-Match" relationship or do some further processing. The Unmatched relationship will contain any FlowFiles where the content for two files of the same filename have content that did not match. The FlowFile will contain the content from SFTP2. Hope this helps. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more