Unable to locate initial admin error when using OIDC in Nifi Cluster mode

[#1]

Unable to up the Nifi Cluster Mode with OIDC Integration

• Have Existing Cert-Manager v1.5.3 (Let's Encrypt)
• Currently using OIDC with Cluster Mode
• Nifi Image version 1.19.1
• Pull nifi release v1.1.3
• Kong gateway/proxy version 3.2

My value.yaml (Note: I only put necessary info value not a full value.yaml)

replicaCount: 2

properties:
  externalSecure: false
  isNode: true
  httpsPort: 8443 
  webProxyHost: nifi-cluster.domain.name:443 

service:
  type: ClusterIP
  httpsPort: 8443
  # nodePort: 30236
  annotations: 
    kubernetes.io/ingress.class: kong-nginx
    konghq.com/protocol: "https"

ingress:
  enabled: true
  className: kong-nginx
  annotations: 
    cert-manager.io/cluster-issuer: letsencrypt-xxx
    konghq.com/strip-path: "false"
    konghq.com/plugins: session
    konghq.com/session-affinity-cookie: my_session_cookie
    konghq.com/protocols: "http,https"  # Enable both HTTP and HTTPS
    konghq.com/session-affinity: "true"  # Enable session affinity
  tls: 
    - secretName: nifi-certificate
      hosts:
        - nifi-cluster.domain.name
  hosts: 
    - nifi-cluster.domain.name
  path: /

My authorizers.xml (Note: my coredns are using cluster.devops)

{{- $replicas := int .Values.replicaCount }}
{{- $chart := .Chart.Name }}
{{- $release := .Release.Name }}
{{- $fullname := include "apache-nifi.fullname" . }}
{{- $namespace := .Release.Namespace }}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at
      http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!--
    This file lists the userGroupProviders, accessPolicyProviders, and authorizers to use when running securely. In order
    to use a specific authorizer it must be configured here and it's identifier must be specified in the nifi.properties file.
    If the authorizer is a managedAuthorizer, it may need to be configured with an accessPolicyProvider and an userGroupProvider.
    This file allows for configuration of them, but they must be configured in order:
    ...
    all userGroupProviders
    all accessPolicyProviders
    all Authorizers
    ...
-->

<authorizers>
    <userGroupProvider>
        <identifier>file-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
        <property name="Users File">./auth-conf/users.xml</property>
        <property name="Legacy Authorized Users File"></property>
        {{- range $i := until $replicas }}
        <property name="Initial User Identity {{ $i }}">CN={{ $fullname }}-{{ $i }}.{{ $fullname }}-headless.{{ $namespace }}.svc.cluster.devops, OU=NIFI</property>
        {{- end }}
    </userGroupProvider>

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">file-user-group-provider</property>
        <property name="Authorizations File">./auth-conf/authorizations.xml</property>
        <property name="Node Identity"></property>
    </accessPolicyProvider>

    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

    {{- if .Values.auth.oidc.enabled}}
    <userGroupProvider>
        <identifier>aad-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.azure.AzureGraphUserGroupProvider</class>
        <property name="Refresh Delay">1 mins</property>
        <property name="Authority Endpoint">https://login.microsoftonline.com</property>
        <property name="Directory ID">{{.Values.auth.oidc.tenantId}}</property>
        <property name="Application ID">{{.Values.auth.oidc.clientId}}</property>
        <property name="Client Secret">{{.Values.auth.oidc.clientSecret}}</property>
        <property name="Group Filter Prefix">aad-nifi</property>
        <property name="Page Size">100</property>
    </userGroupProvider>

    <userGroupProvider>
        <identifier>composite-configurable-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
        <property name="Configurable User Group Provider">file-user-group-provider</property>
        <property name="User Group Provider 1">aad-user-group-provider</property>
    </userGroupProvider>

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">composite-configurable-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">{{.Values.auth.oidc.admin}}</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1"></property>
    </accessPolicyProvider>

    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">aad-user-group-provider</property>
    </authorizer>
    {{- end}}
</authorizers>

Error hit and the nifi pod were in crashloop state

Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies

org.apache.nifi.NiFi Application Server shutdown started

Expected Result

• Able to login in cluster mode(2 Replica) with oidc integration (AzureAD)

Appreciate that someone could help me to find where the configuration needs to be correct.

[#2]

@EddyChan Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @TimothySpann @SAMSAL @steven-matison  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@EddyChan 

In your authorizers.xml it appears you have configured your "managed-authorizer" --> "Access Policy Provider" property to reference a User Group Provider (aad-user-group-provider" instead of an Access Policy Provider.  It should be configured to use the "file-access-policy-provider".

The file-access-policy-provider then gets a list of users and groups from the then referenced user-group-providers in order to generate initial admin access policies.

I see your file-access-policy-provider "initial admin identity" property is configured with:

{{.Values.auth.oidc.admin}}

I don't know what user identity string this resolves to.  Keep in mind that NiFi users and group idnetity strings are case sensitive. So if your admin user identity returned by the "aad-user-group-provider" is "joseace@azure.com" for example and "{{.Values.auth.oidc.admin}}" resolves to "joseAce@azure.com", NiFi recognizes these as two different users.

You may also want to try putting the  add-user-group-provider class in DEBUG in the NiFi logback.xml to see if it provides a listing of all the users and groups returned by that provider configuration.

org.apache.nifi.authorization.azure.AzureGraphUserGroupProvider

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#4]

Hi @MattWho ,

Tested in my env by configure to "file-access-policy-provider" is workable in one replica. But when try two replicas it hit the error like below

user-app log

INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: joseAce@azure.com
INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Sun Jun 04 03:30:39 MYT 2023
INFO [main] o.a.n.a.FileAccessPolicyProvider Added mapped node OU=NIFI, CN=nifi-infra-1.nifi-infra-headless.jose-env.svc.cluster.devops (raw node identity OU=NIFI, CN=nifi-infra-1.nifi-infra-headless.jose-env.svc.cluster.devops)
INFO [main] o.a.n.a.FileAccessPolicyProvider Added mapped node OU=NIFI, CN=nifi-infra-0.nifi-infra-headless.jose-env.svc.cluster.devops (raw node identity OU=NIFI, CN=nifi-infra-0.nifi-infra-headless.jose-env.svc.cluster.devops)
DEBUG [main] o.a.n.a.FileAccessPolicyProvider Empty node group name provided

app-log

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration': Unsatisfied dependency expressed through method 'setObjectPostProcessor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin jose@silverlakeaxis.com to seed policies
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:768)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:720)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:410)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:330)
        ... 67 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:410)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:1027)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:907)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:637)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:583)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:550)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:265)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1557)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1354)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:760)
        ... 87 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1898)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1284)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:345)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
        ... 109 common frames omitted
Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies
        at org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:267)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.nifi.authorization.AccessPolicyProviderInvocationHandler.invoke(AccessPolicyProviderInvocationHandler.java:54)
        at com.sun.proxy.$Proxy81.onConfigured(Unknown Source)
        at org.apache.nifi.authorization.AuthorizerFactoryBean.loadProviderProperties(AuthorizerFactoryBean.java:211)
        at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:168)
        at org.apache.nifi.authorization.AuthorizerFactoryBean.getObject(AuthorizerFactoryBean.java:72)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
        ... 119 common frames omitted
Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies
        at org.apache.nifi.authorization.FileAccessPolicyProvider.populateInitialAdmin(FileAccessPolicyProvider.java:678)
        at org.apache.nifi.authorization.FileAccessPolicyProvider.load(FileAccessPolicyProvider.java:607)
        at org.apache.nifi.authorization.FileAccessPolicyProvider.onConfigured(FileAccessPolicyProvider.java:258)
        ... 129 common frames omitted
INFO [Thread-1] org.apache.nifi.NiFi Application Server shutdown started

Thanks,

Jose

[#5]

@EddyChan 

The exception:

Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin joseAce@azure.com to seed policies

means that on startup the file-access-policy-provider attempted to generate the authorizations.xml file to seed it with initial admin policies for user identity string "joseAce@azure.com"; however, neither of the file-user-group-provider or the aad-user-group-provider returned that user identity string.

So you'll need to take a look at the generated authorizers.xml files when you create your two replicas.  Something is getting messed up in the configuration file.   The order in which the providers are added to the authorizers.xml is very important (User-group-providers must come first, then composite-providers, then file-access-policy-provider, and finally the authorizer).

Perhaps in your file-access-policy-provider, you also have the following property being added which is blank:

<property name="Node Group">nifi</property 

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#6]

@EddyChan, Has any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

[#7]

Tried follow accordingly, not working at the moment hit the same issue. Will look into other configuration.