NIFI-1.16.0 insufficient permissions while performing any action

[#1]

HI Team,

System Env Details: I am trying setup NIFI on AWS
3 node cluster
NIFI version:  1.16.0

authorisers.xml :

<userGroupProvider>
        <identifier>file-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
        <property name="Users File">./conf/users.xml</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Initial User Identity 1">xxx@xxx.com</property>
        <property name="Initial User Identity 2">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_1%%</property>
        <property name="Initial User Identity 3">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_2%%</property> 
        <property name="Initial User Identity 4">C=US, ST=xxxxx, O=xxxx, OU=xxxx, CN=%%cn_user_identity_3%%</property>
    </userGroupProvider>


 <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">file-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">xxxx@xxxxx.com</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1">C=US, ST=xxx, O=xxx, OU=xxx, CN=%%cn_node_identity_1%%</property>
        <property name="Node Identity 2">C=US, ST=xxx, O=xx, OU=xxx, CN=%%cn_node_identity_2%%</property>
        <property name="Node Identity 3">C=US, ST=xxxx, O=xxxxx, OU=xxxx, CN=%%cn_node_identity_3%%</property>
        <property name="Node Group"></property>
    </accessPolicyProvider>

nifi.properties

# Core Properties #
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.json.file=./conf/flow.json.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components

####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# H2 Settings
nifi.database.directory=%%base_path%%/database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

# Repository Encryption properties override individual repository implementation properties
nifi.repository.encryption.protocol.version=
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=%%base_path%%/flowfile_repository
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.repository.directory.default=%%base_path%%/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=%%base_path%%/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable.  Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2


# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component and Node Status History Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository

# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# QuestDB Status History Repository Properties
nifi.status.repository.questdb.persist.node.days=14
nifi.status.repository.questdb.persist.component.days=3
nifi.status.repository.questdb.persist.location=%%base_path%%/status_repository

# Site to Site properties
nifi.remote.input.host=%%host_name%%
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=

#############################################

nifi.web.https.host=%%host_name%%
nifi.web.https.port=8086
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=%%proxy_hostname%%:443
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

# Include or Exclude TLS Cipher Suites for HTTPS
nifi.web.https.ciphersuites.include=
nifi.web.https.ciphersuites.exclude=

# security properties #
nifi.sensitive.props.key=***************
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=%%base_path%%/certs/SSL/%%host_name%%.keystore
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=**********
nifi.security.keyPasswd=*********
nifi.security.truststore=%%base_path%%/certs/SSL/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=***********
nifi.security.user.authorizer=managed-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=
nifi.security.user.jws.key.rotation.period=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=

# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=
nifi.security.user.oidc.fallback.claims.identifying.user=
nifi.security.user.oidc.truststore.strategy=JDK

# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# SAML Properties #
nifi.security.user.saml.idp.metadata.url=file://%%base_path%%/certs/IDMS/idms-xxxxx.xml
nifi.security.user.saml.sp.entity.id=xxxxxxx
nifi.security.user.saml.identity.attribute.name=email
nifi.security.user.saml.group.attribute.name=
nifi.security.user.saml.metadata.signing.enabled=false
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.signature.digest.algorithm=http://www.w3.org/2001/04/xmlenc#sha256
nifi.security.user.saml.message.logging.enabled=true
nifi.security.user.saml.authentication.expiration=8 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER

# Listener Bootstrap properties #
# This property defines the port used to listen for communications from NiFi Bootstrap. If this property
# is missing, empty, or 0, a random ephemeral port is used.
nifi.listener.bootstrap.port=0

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=%%host_name%%
nifi.cluster.node.protocol.port=11443
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=10 sec
nifi.cluster.node.read.timeout=10 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=%%zookeeper_connection_strings%%
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=%%base_path%%/nifi
nifi.zookeeper.client.secure=true
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.jute.maxbuffer=

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=

# analytics properties #
nifi.analytics.predict.enabled=false
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90

# runtime monitoring properties
nifi.monitor.long.running.task.schedule=
nifi.monitor.long.running.task.threshold=

# Create automatic diagnostics when stopping/restarting NiFi.

# Enable automatic diagnostic at shutdown.
nifi.diagnostics.on.shutdown.enabled=false

# Include verbose diagnostic information.
nifi.diagnostics.on.shutdown.verbose=false

# The location of the diagnostics folder.
nifi.diagnostics.on.shutdown.directory=./diagnostics

# The maximum number of files permitted in the directory. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.filecount=10

# The diagnostics folder's maximum permitted size in bytes. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.directory.size=10 MB

Users.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups>
        <group identifier="39903356-0181-1000-0000-000047f790c3" name="Admin">
            <user identifier="3989a18a-0181-1000-0000-0000280a357e"/>
            <user identifier="5c72646f-a9cb-3239-9450-2511231b004e"/>
            <user identifier="6d337594-bfba-3a95-88a7-01ec124f0bdb"/>
            <user identifier="a98e60ec-bb80-35c6-bd31-1d1c7562a5a8"/>
            <user identifier="2e8a3365-86df-3c2d-8614-7cf2c61cc5d1"/>
        </group>
    </groups>
    <users>
        <user identifier="2e8a3365-86df-3c2d-8614-7cf2c61cc5d1" identity="C=US, ST=xxx, O=xxx, OU=xxx, CN=xxxx"/>
        <user identifier="a98e60ec-bb80-35c6-bd31-1d1c7562a5a8" identity="C=US, ST=xxx, O=xxx, OU=xxx, CN=xxx"/>
        <user identifier="5c72646f-a9cb-3239-9450-2511231b004e" identity="xxx@xxx.com"/>
        <user identifier="6d337594-bfba-3a95-88a7-01ec124f0bdb" identity="C=US, ST=xxx, O=xxx, OU=xxxx, CN=xxxxx"/>
        <user identifier="3989a18a-0181-1000-0000-0000280a357e" identity="abc@abc.com"/>
    </users>
</tenants>

I am using SAML and successfully able to login as an Initial Admin Identity. However after login I can see my UI as seen in image, where all he processors are disabled for me.



And if I perform any action like logout or add/edit user, add policy.... I get below error message:



I have similar setup with NIFI version 1.10.0, and it is working fine. But with Nifi-1.15.0 or Nifi-1.16.0 I am unable to perform any action.

Any help would be highly appreciated.

[#2]

Hi @araujo, 

Have you experience it ? Request your help.

Thanks

[#3]

Can you share a snapshot of nifi-user.log file captured when you get 403 error?

[#4]

HI @gtorres,

I am not able to see any 403 activity even getting logged in my servers. I have not found any such 403 request/response calls in  App logs, user logs, Request logs.

Whenever I am getting 403 while performing any actions, it is not getting captured in any log file (App, User, Request).

https://community.cloudera.com/t5/Support-Questions/Unable-to-seed-access-policy-on-NIFI-1-16-0-and-...

[#5]

@Abhishek27Apple 
Since you are not seeing anything in the NiFi log files...
1. Have you tried using a different web browser like Firefox?
2. Have you tried opening your browser's Developer tools and inspecting the actual rest-api call that was made when you attempt the various actions that fail from with the NiFi UI?
3. Are you going through a proxy or load balancer (is it configured to use sticky sessions?)?
4. Which Browser and version are you using?
5. Have you tried clearing your browser cache?
6. Does same behavior exist using an incognito window in your browser?
7. What java version is your NiFi using?

Thank you,
Matt

[#6]

@MattWho 

1. Have you tried using a different web browser like Firefox?

I am unable to log in via Mozilla, my SAML service is not working for callbacks to NIFI.
However, used Safari and Chrome.

2. Have you tried opening your browser's Developer tools and inspecting the actual rest-api call that was made when you attempt the various actions that fail from with the NiFi UI?

Yes, I did it. I am sharing details for few of the actions:

Request URL: https://xxxxxxxx/nifi-api/policies/43dfca36-0181-1000-ffff-ffff90447006
Request Method: PUT
Status Code: 403

Payload: {"revision":{"clientId":"4806d375-0181-1000-e24c-c2e2244d8578","version":0},"disconnectedNodeAcknowledged":false,"component":{"id":"43dfca36-0181-1000-ffff-ffff90447006","users":[{"revision":{"version":0},"id":"5c72646f-a9cb-3239-9450-2511231b004e","permissions":{"canRead":true,"canWrite":true},"component":{"id":"5c72646f-a9cb-3239-9450-2511231b004e","identity":"xxxx@xxx.com","configurable":true}}],"userGroups":[]}}

Request URL: https://xxxxxxxxx/nifi-api/flow/process-groups/439aeba6-0181-1000-fabf-28c9c87d5ce8/controller-servi...
Request Method: PUT
Status Code: 403

Payload: {"id":"439aeba6-0181-1000-fabf-28c9c87d5ce8","state":"ENABLED","disconnectedNodeAcknowledged":false}

https://xxxxxxx/nifi-api/access/logout
Request Method: DELETE
Status Code: 403

3. Are you going through a proxy or load balancer (is it configured to use sticky sessions?)?

Yes, I am using a proxy and have configured sticky sessions.

4. Which Browser and version are you using?

Safari: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15

Chrome: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36

For both, the browser NIFI adds Mozilla/5.0 in user-agent

5. Have you tried clearing your browser cache?

- Yes I tried clearing my cache, It exhibits the same behavior.

6. Does the same behavior exist using an incognito window in your browser?

I have tried on Safari, and Chrome. However, I have used Safari private window as well. It exhibits the same behavior.

However unable to Logging using Chrome- incognito window.

7. What java version is your NiFi using?
- JDK-11.0.14.9.2


In addition to these, I would like to bring you to notice that, I have tried NIFI- 1.16.0 on AWS with a single node setup, using single-user-authorizer. It works absolutely fine. And I am able to use NIFI smoothly. Once I tried using managed-authorizer with SAML on 3 node cluster. Then I am facing all these permission issues.

However, I have an absolutely similar setup with 3 node cluster for NIFI-1.150 (not on AWS, it is on the private cloud). There I am able to get access to the main root process policy. And able to access processors can create processors, but can not delete processors, Logout and add policy for other users or myself.

[#7]

@MattWho @gtorres @SAMSAL 

I am sharing my user logs. 

2022-06-15 04:18:06,090 DEBUG [NiFi Web Server-18] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,090 DEBUG [NiFi Web Server-18] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-06-15 04:18:06,091 DEBUG [NiFi Web Server-18] o.a.n.w.s.j.k.StandardVerificationKeySelector Key Identifier [ec9c28d4-7330-48da-bdf5-dd398cd5b76f] Verification Keys Found [1]
2022-06-15 04:18:06,093 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,093 DEBUG [NiFi Web Server-18] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,093 DEBUG [NiFi Web Server-143] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-06-15 04:18:06,093 DEBUG [NiFi Web Server-18] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,093 DEBUG [NiFi Web Server-143] o.a.n.w.s.j.k.StandardVerificationKeySelector Key Identifier [ec9c28d4-7330-48da-bdf5-dd398cd5b76f] Verification Keys Found [1]
2022-06-15 04:18:06,094 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,094 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-128] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***>
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-128] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:18:06,108 INFO [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=******.***>] GET https://********.***.**.***:8086/nifi-api/flow/status
2022-06-15 04:18:06,108 INFO [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [********@Company.***] 10.204.230.155 GET https://********.***.**.***:8086/nifi-api/flow/status
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-136] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***>
2022-06-15 04:18:06,108 DEBUG [NiFi Web Server-136] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:18:06,108 INFO [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=******.***>] GET https://********.***.**.***:8086/nifi-api/flow/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:18:06,109 INFO [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [********@Company.***] 10.204.230.155 GET https://********.***.**.***:8086/nifi-api/flow/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:18:06,109 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,109 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,359 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,359 DEBUG [NiFi Web Server-136] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-06-15 04:18:06,359 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,360 DEBUG [NiFi Web Server-128] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-06-15 04:18:06,360 DEBUG [NiFi Web Server-136] o.a.n.w.s.j.k.StandardVerificationKeySelector Key Identifier [ec9c28d4-7330-48da-bdf5-dd398cd5b76f] Verification Keys Found [1]
2022-06-15 04:18:06,360 DEBUG [NiFi Web Server-128] o.a.n.w.s.j.k.StandardVerificationKeySelector Key Identifier [ec9c28d4-7330-48da-bdf5-dd398cd5b76f] Verification Keys Found [1]
2022-06-15 04:18:06,361 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,361 DEBUG [NiFi Web Server-143] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-06-15 04:18:06,362 DEBUG [NiFi Web Server-143] o.a.n.w.s.j.k.StandardVerificationKeySelector Key Identifier [ec9c28d4-7330-48da-bdf5-dd398cd5b76f] Verification Keys Found [1]
2022-06-15 04:18:06,362 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,362 DEBUG [NiFi Web Server-136] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,362 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,362 DEBUG [NiFi Web Server-128] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,363 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,363 DEBUG [NiFi Web Server-143] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,373 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,373 DEBUG [NiFi Web Server-20] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***>
2022-06-15 04:18:06,373 DEBUG [NiFi Web Server-20] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:18:06,373 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=******.***>] GET https://********.***.**.***:8086/nifi-api/flow/current-user
2022-06-15 04:18:06,373 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [********@Company.***] 10.204.230.155 GET https://********.***.**.***:8086/nifi-api/flow/current-user
2022-06-15 04:18:06,373 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,373 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,378 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:18:06,378 DEBUG [NiFi Web Server-20] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***>
2022-06-15 04:18:06,378 DEBUG [NiFi Web Server-20] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:18:06,378 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<********@Company.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=********.***.**.***><C=US, ST=California, O=Company ***, OU=********.***.**.***, CN=******.***>] GET https://********.***.**.***:8086/nifi-api/flow/controller/bulletins
2022-06-15 04:18:06,378 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [********@Company.***] 10.204.230.155 GET https://********.***.**.***:8086/nifi-api/flow/controller/bulletins
2022-06-15 04:18:06,379 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]
2022-06-15 04:18:06,379 DEBUG [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [********@Company.***]

2022-06-15 04:16:23,463 DEBUG [NiFi Web Server-120] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:16:23,463 INFO [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<xxxxxxxxx@company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=zzzzzzzzzz.company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=yyyyyyyy.company.com>] GET https://xxxxxxxxx.company.com:8086/nifi-api/policies/write/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:16:23,464 INFO [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [xxxxxxxxx@company.com] 10.204.230.155 GET https://xxxxxxxxx.company.com:8086/nifi-api/policies/write/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:16:23,464 DEBUG [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:16:23,464 DEBUG [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:16:27,281 DEBUG [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:16:27,281 DEBUG [NiFi Web Server-21] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <xxxxxxxxx@company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=zzzzzzzzzz.company.com>
2022-06-15 04:16:27,281 DEBUG [NiFi Web Server-21] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:16:27,281 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<xxxxxxxxx@company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=zzzzzzzzzz.company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=yyyyyyyy.company.com>] GET https://xxxxxxxxx.company.com:8086/nifi-api/policies/write/operation/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:16:27,282 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [xxxxxxxxx@company.com] 10.204.230.155 GET https://xxxxxxxxx.company.com:8086/nifi-api/policies/write/operation/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
2022-06-15 04:16:27,282 DEBUG [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:16:27,282 DEBUG [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:16:27,302 INFO [NiFi Web Server-21] o.a.n.w.a.c.ResourceNotFoundExceptionMapper org.apache.nifi.web.ResourceNotFoundException: Unable to find access policy for write on /operation/process-groups/653f47d8-0181-1000-1d99-58b6a323962a. Returning Not Found response.
2022-06-15 04:16:27,309 DEBUG [NiFi Web Server-21] o.a.n.w.a.c.ResourceNotFoundExceptionMapper 
org.apache.nifi.web.ResourceNotFoundException: Unable to find access policy for write on /operation/process-groups/653f47d8-0181-1000-1d99-58b6a323962a
	at org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO.getAccessPolicy(StandardPolicyBasedAuthorizerDAO.java:201)
	at org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO$$FastClassBySpringCGLIB$$ea190383.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
	at org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO$$EnhancerBySpringCGLIB$$be10ced9.getAccessPolicy(<generated>)
	at org.apache.nifi.web.StandardNiFiServiceFacade.getAccessPolicy(StandardNiFiServiceFacade.java:4089)
	at org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:161)
	at org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:120)
	at jdk.internal.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
	at org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$26e6223b.getAccessPolicy(<generated>)
	at org.apache.nifi.web.api.AccessPolicyResource.getAccessPolicyForResource(AccessPolicyResource.java:162)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
	at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.oauth2.server.resource.web.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:121)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:94)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:56)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
	at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
	at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022-06-15 04:16:31,701 DEBUG [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]
2022-06-15 04:16:31,701 DEBUG [NiFi Web Server-120] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <xxxxxxxxx@company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=zzzzzzzzzz.company.com>
2022-06-15 04:16:31,701 DEBUG [NiFi Web Server-120] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-06-15 04:16:31,701 INFO [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.204.230.155 [<xxxxxxxxx@company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=zzzzzzzzzz.company.com><C=US, ST=California, O=Company, OU=xxxxxxxxx, CN=yyyyyyyy.company.com>] POST https://xxxxxxxxx.company.com:8086/nifi-api/policies
2022-06-15 04:16:31,701 INFO [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [xxxxxxxxx@company.com] 10.204.230.155 POST https://xxxxxxxxx.company.com:8086/nifi-api/policies

user.logs

Here we have observed that whenever NIFI tries to get status or fetch the current user, we get below mentioned logs, weherNiFiAuthenticationFilter Authenticating sometimes gets the user and sometimes log with null:

10.204.230.155 GET https://xxxxxxxxx.company.com:8086/nifi-api/flow/status
2022-06-15 04:14:57,583 DEBUG [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:14:57,583 DEBUG [NiFi Web Server-25] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:14:57,583 DEBUG [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:14:57,583 DEBUG [NiFi Web Server-25] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [xxxxxxxxx@company.com]
2022-06-15 04:14:57,818 DEBUG [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authenticating [null]

[#8 · Accepted solution]

Thank you everyone for all the support and help.

RCA: What I have observed, whenever we are performing PUT or Delete HTTP requests, the proxy in front of NiFi is intercepting and denying the request. Hence these requests are not even reaching our EC2 instance, and NIFI request.log is not capturing any PUT/DELETE HTTP requests. Hence on UI, we are getting 403 permissions Issues.

Solution: We have enabled Put/ Delete HTTP requests from the proxy, and are now able to perform all the actions.

Hence we can close this ticket.

[#9]

How did anyone solve this issue? I was currently trying to solve similar issue via nifi api (Postman)