nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack

obrobecker — Sat, 23 Feb 2019 03:51:02 GMT

hello,

We have installed a secured hdp 3.1 cluster on Centos 7.5.

Then we installed mpack in order to add a nifi single node.

The unsecure version worked correctly (at least it displayed the ui correctly ) but upon activating ssl ( with auto generated certificate ) and activating kerberos for authentification, when connecting, we got the following error :

Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not connected

This is strange because we use the secure version and connect to nifi via https://my_nifi_FDQN_node:9091/nifi/ and it should not try to connect to 9090.

In nifi-user.log, we can see :

2019-02-22 11:19:10,767 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for my_ldap_user
2019-02-22 11:19:10,772 INFO [NiFi Web Server-21] o.a.n.w.a.c.IllegalClusterStateExceptionMapper org.apache.nifi.cluster.manager.exception.IllegalClusterStateException: Cannot replicate request to Node my_nifi_FDQN_node:9090 because the node is not connected. Returning Conflict response.

I don't know if it has something to do with it but I also got the following audit-log error in the nifi-app.log

2019-02-22 11:19:08,991 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2019-02-22 11:19:08,861 and sent to my_nifi_FDQN_node:9088 at 2019-02-22 11:19:08,991; send took 130 millis
2019-02-22 11:19:11,865 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler Audit Status Log: name=nifi.async.batch.hdfs, interval=11:42.021 minutes, events=1, deferredCount=1, totalEvents=5, totalDeferredCount=5
2019-02-22 11:19:11,866 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.a.destination.HDFSAuditDestination Returning HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml
2019-02-22 11:19:11,879 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.a.destination.HDFSAuditDestination Checking whether log file exists. hdfPath=hdfs://my_master_node:8020/ranger/audit/nifi/20190222/nifi_ranger_audit_my_nifi_FDQN_node.log, UGI=nifi/_HOST@REALM (auth:KERBEROS)
2019-02-22 11:19:11,887 ERROR [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler Error writing to log file.
java.io.IOException: DestHost:destPort my_master_node:8020 , LocalHost:localPort my_nifi_FDQN_node/my_nifi_IP_node:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
2019-02-22 11:19:11,887 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.a.destination.HDFSAuditDestination Flushing HDFS audit. Event Size:1
2019-02-22 11:19:11,887 WARN [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler failed to log audit event: {"repoType":10,"repo":"datalake_prod_nifi","reqUser":"XXXX","evtTime":"2019-02-22 11:19:10.770","access":"READ","resource":"/flow","resType":"nifi-resource","action":"READ","result":1,"policy":18,"enforcer":"ranger-acl","cliIP":"client_ip","agentHost":"my_nifi_FDQN_node","logType":"RangerAudit","id":"cf2fd979-945c-4461-a1df-c40c42defdd1-5","seq_num":11,"event_count":1,"event_dur_ms":0,"tags":[]}, errorMessage=
2019-02-22 11:19:11,887 WARN [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler Log failure count: 1 in past 11:42.022 minutes; 6 during process lifetime

Nifi is very new to me so I'm not sure what information to look for.

BR,

Re: nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack

obrobecker — Tue, 26 Feb 2019 19:34:32 GMT

Hi,

Found out that there was a snapshot file hanging around with a reference to the old unsecure URL.

I've deleted the /var/lib/nifi/state/local/snapshot file and I nearly works. got an authorization error but some ranger tuning will overcomes it.

Re: nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack

poom — Thu, 14 Mar 2019 14:51:22 GMT

I've this error after rollback from SSL config
"nifi Cannot replicate request to Node because the node is not connected"

It's Work.

Thanks.

question Re: nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack in Support Questions

nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack

Re: nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack

Re: nifi issue accessing UI after activating kerberos with ranger on HDP3.1 with mpack