Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Receiving continuous alerts in Nifi-registry log files - Failed to specify server's Kerberos principal name

Highlighted

Receiving continuous alerts in Nifi-registry log files - Failed to specify server's Kerberos principal name

New Contributor

Receiving continuous alerts in Nifi-registry log files - Failed to specify server's Kerberos principal name


Cluster Specifications:

HDP 3.1

HDF 3.4

Kerberos enabled

Ranger SSL and PLUGINS enabled for all services

All services are green up and running.

Only issue is the below log.


tail -f nifi-registry-app.log

2019-04-29 10:31:18,219 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Audit Status Log: name=Testcluster_nifi_registry.async.multi_dest.batch.hdfs, interval=01:00.007 minutes, events=1, deferredCount=1, totalEvents=21, totalDeferredCount=21

2019-04-29 10:31:18,219 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Returning HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml

2019-04-29 10:31:18,222 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Checking whether log file exists. hdfPath=hdfs://horton-mgmt1.XXXXXXX.local:8020/ranger/audit/Testcluster_nifi_registry/20190429/Testcluster_nifi_registry_ranger_audit_horton-hdf0.XXXXXXX.local.log, UGI=nifiregistry/horton-hdf0.XXXXXXX.local@XXXXXXX.LOCAL (auth:KERBEROS)

2019-04-29 10:31:18,225 ERROR [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file.

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "horton-hdf0.XXXXXXX.local/xx.xx.xx.xx"; destination host is: "horton-mgmt1.XXXXXXX.local":8020;

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0.jar:na]

at com.sun.proxy.$Proxy128.getFileInfo(Unknown Source) ~[na:na]

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:874) ~[hadoop-hdfs-client-3.0.0.jar:na]

at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112]

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0.jar:na]

at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source) ~[na:na]

at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1697) ~[hadoop-hdfs-client-3.0.0.jar:na]

at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0.jar:na]

at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1488) ~[hadoop-hdfs-client-3.0.0.jar:na]

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1503) ~[hadoop-hdfs-client-3.0.0.jar:na]

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0.jar:na]

at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:289) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112]

at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na]

at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) [ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) [ranger-plugins-audit-1.1.0.jar:1.1.0]

at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) [ranger-plugins-audit-1.1.0.jar:1.1.0]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]

Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:860) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client.call(Client.java:1383) ~[hadoop-common-3.0.0.jar:na]

... 33 common frames omitted

Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name

at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:613) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:409) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794) ~[hadoop-common-3.0.0.jar:na]

at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112]

at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na]

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794) ~[hadoop-common-3.0.0.jar:na]

... 36 common frames omitted

2019-04-29 10:31:18,225 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Flushing HDFS audit. Event Size:1

2019-04-29 10:31:18,225 ERROR [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.ranger.audit.queue.AuditFileSpool Error sending logs to consumer. provider=Testcluster_nifi_registry.async.multi_dest.batch, consumer=Testcluster_nifi_registry.async.multi_dest.batch.hdfs

2019-04-29 10:31:18,225 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.ranger.audit.queue.AuditFileSpool Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=Testcluster_nifi_registry.async.multi_dest.batch, consumer=Testcluster_nifi_registry.async.multi_dest.batch.hdfs


NiFi registry configurations :

Advanced nifi-registry-authorizers-env

<authorizers>

<userGroupProvider>

<identifier>ldap-user-group-provider</identifier>

<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>

<property name="Authentication Strategy">LDAPS</property>

<property name="Identity Strategy">USE_USERNAME</property>

<property name="Manager DN">CN=xxx,OU=xx,OU=xx,DC=xxx,DC=xxx</property>

<property name="Manager Password">xxxxxx</property>

<property name="TLS - Keystore">/etc/nifi-registry/conf/certs_stores/nifi-registry-keystore.jks</property>

<property name="TLS - Keystore Password"> xxxxxx </property>

<property name="TLS - Keystore Type">JKS</property>

<property name="TLS - Truststore">/etc/nifi-registry/conf/certs_stores/nifi-registry-truststore.jks</property>

<property name="TLS - Truststore Password"> xxxxxx </property>

<property name="TLS - Truststore Type">JKS</property>

<property name="TLS - Client Auth">WANT</property>

<property name="TLS - Protocol">TLS</property>

<property name="TLS - Shutdown Gracefully"></property>

<property name="Referral Strategy">FOLLOW</property>

<property name="Connect Timeout">10 secs</property>

<property name="Read Timeout">10 secs</property>

<property name="Url">ldaps:// xxxxxx:636</property>

<property name="Page Size"></property>

<property name="Sync Interval">30 mins</property>

<property name="User Search Base"></property>

<property name="User Object Class">person</property>

<property name="User Search Scope">ONE_LEVEL</property>

<property name="User Search Filter">sAMAccountName={0}</property>

<property name="User Identity Attribute"></property>

<property name="User Group Name Attribute"></property>

<property name="User Group Name Attribute - Referenced Group Attribute"></property>

<property name="Group Search Base">DC= xxxxxx,DC= xxxxxx</property>

<property name="Group Object Class">group</property>

<property name="Group Search Scope">ONE_LEVEL</property>

<property name="Group Search Filter"></property>

<property name="Group Name Attribute">cn</property>

<property name="Group Member Attribute">member</property>

<property name="Group Member Attribute - Referenced User Attribute"></property>


</userGroupProvider>

<authorizer>

<identifier>ranger-authorizer</identifier>

<class>org.apache.nifi.registry.ranger.RangerAuthorizer</class>

<property name="User Group Provider">ldap-user-group-provider</property>

<property name="Ranger Audit Config Path">/etc/nifi-registry/conf/ranger-nifi-registry-audit.xml</property>

<property name="Ranger Security Config Path">/etc/nifi-registry/conf/ranger-nifi-registry-security.xml</property>

<property name="Ranger Service Type">nifi-registry</property>

<property name="Ranger Application Id">Testcluster_nifi_registry</property>

<property name="Ranger Admin Identity">CN= xxxxxx,OU= xxxxxx,OU= xxxxxx,DC= xxxxxx,DC= xxxxxx </property>

<property name="Ranger Kerberos Enabled">true</property>

</authorizer>

</authorizers>



Advanced nifi-registry-identity-providers-env

<identityProviders>

<provider>

<identifier>ldap-identity-provider</identifier>

<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>

<property name="Authentication Strategy">LDAPS</property>

<property name="Identity Strategy">USE_USERNAME</property>

<property name="Manager DN">CN=xxxxxx,OU= xxxxxx,OU= xxxxxx,DC= xxxxxx,DC= xxxxxx </property>

<property name="Manager Password"> xxxxxx </property>

<property name="TLS - Keystore">/etc/nifi-registry/conf/certs_stores/nifi-registry-keystore.jks</property>

<property name="TLS - Keystore Password"> xxxxxx </property>

<property name="TLS - Keystore Type">JKS</property>

<property name="TLS - Truststore">/etc/nifi-registry/conf/certs_stores/nifi-registry-truststore.jks</property>

<property name="TLS - Truststore Password"> xxxxxx </property>

<property name="TLS - Truststore Type">JKS</property>

<property name="TLS - Client Auth">NONE</property>

<property name="TLS - Protocol">TLS</property>

<property name="TLS - Shutdown Gracefully"></property>

<property name="Referral Strategy">FOLLOW</property>

<property name="Connect Timeout">10 secs</property>

<property name="Read Timeout">10 secs</property>

<property name="Url">ldaps:// xxxxxx:636</property>

<property name="User Search Base">DC= xxxxxx,DC= xxxxxx </property>

<property name="User Search Filter">sAMAccountName={0}</property>

<property name="Authentication Expiration">12 hours</property>

</provider>

</identityProviders>