Unknown user with identity OIDC Azure AD

[#1]

Hello, i'm using the cetic helm chart to deploy nifi to GCP gke cluster. I have configured SSL and OIDC with azure AD, i was able to see the AD page and log in, after that i'm seeing this error 

Values:

replicaCount: 1


image:
  repository: apache/nifi
  tag: "1.16.3"
  pullPolicy: "IfNotPresent"

securityContext:
  runAsUser: 1000
  fsGroup: 1000


sts:
  podManagementPolicy: Parallel
  AntiAffinity: soft
  useHostNetwork: null
  hostPort: null
  pod:
    annotations:
      security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
      #prometheus.io/scrape: "true"
  serviceAccount:
    create: false
    #name: nifi
    annotations: {}
  hostAliases: []

  startupProbe:
    enabled: false
    failureThreshold: 60
    periodSeconds: 10

properties:
  sensitiveKey: changeMechangeMe 
  externalSecure: true
  isNode: false
  httpsPort: 8443
  httpPort: 8080
  webProxyHost: nifi-test.qa.cloud.paycomplete.com
  clusterPort: 6007
  provenanceStorage: "8 GB"
  siteToSite:
    port: 10000
 configuration
  safetyValve:
    nifi.web.http.network.interface.default: eth0



# Nifi User Authentication
auth:
  # If set while LDAP is enabled, this value will be used for the initial admin and not the ldap bind dn / admin
  admin: CN=axel.fulop@paycomplete.com
  SSL:
    keystorePasswd: changemechangemenifiqa
    truststorePasswd: changemechangemenifiqa
  oidc:
    enabled: true
    discoveryUrl: https://login.microsoftonline.com/DIRECTORYID/v2.0/.well-known/openid-configuration
    clientId: MYCLIENT
    clientSecret: MYSECRET
    claimIdentifyingUser: upn
    admin: axel.fulop@paycomplete.com
    additionalScopes: profile, email, openid


# headless service
headless:
  type: ClusterIP
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"

# ui service
service:
  type: ClusterIP
  httpsPort: 8443
  httpPort: 8080
  #nodePort: 30236
  annotations: 
    # external-dns.alpha.kubernetes.io/hostname: nifi-test.qa.cloud.paycomplete.com
    # loadBalancerIP:
    ## Load Balancer sources
    ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
    ##
    # loadBalancerSourceRanges:
    # - 10.10.10.0/24
    ## OIDC authentication requires "sticky" session on the LoadBalancer for JWT to work properly...but AWS doesn't like it on creation
    # sessionAffinity: ClientIP
    # sessionAffinityConfig:
    #   clientIP:
  #     timeoutSeconds: 10800

  # Enables additional port/ports to nifi service for internal processors
  processors:
    enabled: false
    ports:
      - name: processor01
        port: 7001
        targetPort: 7001
        #nodePort: 30701
      - name: processor02
        port: 7002
        targetPort: 7002
        #nodePort: 30702

## Configure Ingress based on the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/
##
ingress:
  enabled: true
  annotations: 
    kubernetes.io/ingress.class: "nginx"	
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"	
    nginx.ingress.kubernetes.io/proxy-redirect-from: "https://localhost:8443"
    nginx.ingress.kubernetes.io/proxy-redirect-to: "https://nifi-test.qa.cloud.paycomplete.com"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/upstream-vhost: localhost:8443
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"    
    cert-manager.io/issuer: letsencrypt-production
  tls: 
  - hosts:
    - nifi-test.qa.cloud.paycomplete.com
    secretName: nifi-tls-secret
  hosts: 
    - nifi-test.qa.cloud.paycomplete.com
  path: /
  # If you want to change the default path, see this issue https://github.com/cetic/helm-nifi/issues/22

nifi.properties:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Core Properties #
nifi.flow.configuration.file=../data/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=../data/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=../data/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.directory.custom=
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components

####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties


# H2 Settings
nifi.database.directory=../data/database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.directory=../flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=../content_repository
nifi.content.repository.archive.max.retention.period=3 days
nifi.content.repository.archive.max.usage.percentage=85%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=../provenance_repository
nifi.provenance.repository.max.storage.time=10 days
nifi.provenance.repository.max.storage.size=8 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable.  Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536

# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# Site to Site properties
nifi.remote.input.host=nifi-0.nifi-headless.apache-nifi.svc.cluster.local
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10000
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.war.directory=./lib
nifi.web.proxy.host=nifi-test.qa.cloud.paycomplete.com
nifi.web.https.port=8443
nifi.web.http.host=nifi-0.nifi-headless.apache-nifi.svc.cluster.local
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
# nifi.web.proxy.context.path=

# security properties #
nifi.sensitive.props.key=changeMechangeMe
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=


nifi.security.keystore=/opt/nifi/nifi-current/conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=64d3945a969994427c9c47272e104ec1
nifi.security.keyPasswd=64d3945a969994427c9c47272e104ec1
nifi.security.truststore=/opt/nifi/nifi-current/conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=b58f12d3ce529ca05c36706ea6ed9475
nifi.security.user.authorizer=managed-authorizer

nifi.security.needClientAuth=


# OpenId Connect SSO Properties #
nifi.initial.admin.identity = axel.fulop@paycomplete.com
nifi.security.user.oidc.discovery.url=https://login.microsoftonline.com/7bc71559-46eb-47fc-a0a7-327d1337592c/v2.0/.well-known/openid-configuration
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=id
nifi.security.user.oidc.client.secret=secret
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.claim.identifying.user=upn
nifi.security.user.oidc.additional.scopes=profile, email, openid



# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=false
nifi.cluster.node.address=nifi-0.nifi-headless.apache-nifi.svc.cluster.local
nifi.cluster.node.protocol.port=6007
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zookeeper:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
nifi.security.user.login.identity.provider=
nifi.web.http.network.interface.lo=lo

Authorizers:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <userGroupProvider>
    <identifier>file-user-group-provider</identifier>
    <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
    <property name="Users File">./auth-conf/users.xml</property>
    <property name="Legacy Authorized Users File"/>
    <property name="Initial User Identity 1">axel.fulop@paycomplete.com</property>
  </userGroupProvider>

  <accessPolicyProvider>
    <identifier>file-access-policy-provider</identifier>
    <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
    <property name="User Group Provider">file-user-group-provider</property>
    <property name="Authorizations File">./auth-conf/authorizations.xml</property>
    <property name="Initial Admin Identity">axel.fulop@paycomplete.com</property>
    <property name="Legacy Authorized Users File"/>
    <property name="Node Identity 1"/>
    <property name="Node Group"/>
  </accessPolicyProvider>

  <authorizer>
    <identifier>managed-authorizer</identifier>
    <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
    <property name="Access Policy Provider">file-access-policy-provider</property>
  </authorizer>
  

</authorizers>

users:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b" identity="axel.fulop@paycomplete.com"/>
    </users>
</tenants>

authorizations:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="9a110a21-cbce-30b3-8b9e-110bbf96a04d" resource="/data/process-groups/9913cc63-f5a1-39a0-ab12-1551d6d72a86" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="957ed60d-9c11-32e0-8efb-28e669b5955a" resource="/data/process-groups/9913cc63-f5a1-39a0-ab12-1551d6d72a86" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="dd2022f3-1adb-31e7-bee1-688a27f455dc" resource="/process-groups/9913cc63-f5a1-39a0-ab12-1551d6d72a86" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="25449c36-7fc2-34b0-bf6a-e094480853c0" resource="/process-groups/9913cc63-f5a1-39a0-ab12-1551d6d72a86" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
        <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
            <user identifier="6b53d661-deec-3bdc-bd00-6c07e763a48b"/>
        </policy>
    </policies>
</authorizations>

I followed this tutorial to create the Azure AD app : https://github.com/benkelly/NiFi-Authentication-with-Azure-Active-Directory-Setup-Guide

Any idea of what i'm missing?

Thanks

[#2 · Accepted solution]

@axelfulop 

NiFi Authorization is case sensitive.  In your authorizers.xml (responsible for generating the users.xml and authorizationsxml files if and only if they do NOT already exist) you have defined your "initial user identity 1" and "Iniital admin identity" as: 

axel.fulop@paycomplete.com

However, your attached screenshot of the the NiFi UI, shows the authenticated user's 'user identity" was:

Axel.Fulop@paycomplete.com

 NiFi's authorizer will treat these two identities as different user identities since it is case sensitive.

There are two ways to resolve this:
1. Update Authorizers.xml entries to use "Axel.Fulop@paycomplete.com", remove users.xml, remove authorizations.xml and then restart NiFi.  These two files will get recreated based on new user identity.

2. Add a set of identity mapping properties to the nifi.properties file that will take all authenticated user identities and set them to all lowercase:

nifi.security.identity.mapping.pattern.username=^(.*?)$
nifi.security.identity.mapping.value.username=$1
nifi.security.identity.mapping.transform.username=LOWER

This would result in the identity string resulting from successful authentication is converted to all lowercase before being passed to the configured NiFi authorizer.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#3]

Hello Matt, it was that! omg, 2 weeks stucked  with this and was just case sensitive!

Thanks!

[#4]

@MattWho  i'm tried with the authorizers of https://github.com/benkelly/NiFi-Authentication-with-Azure-Active-Directory-Setup-Guide to enabled groups i had created in Azure AD but i'm getting this error:

Error creating bean with name 'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException

at org.apache.nifi.NiFi.main(NiFi.java:330)
2023-04-07 20:32:52,156 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Fri Apr 07 20:32:52 UTC 2023
2023-04-07 20:32:54,832 ERROR [main] o.a.n.a.AuthorizerFactoryBean User Group Provider [composite-configurable-user-group-provider] destruction failed
java.lang.NullPointerException: null

Any idea of what it's going wrong?

[#5]

I updated the Nifi version to the latest and worked!