Issue : Ambari server logs throwing errors in log file after enabling HDFS SSL Enviornment: HDP 3.1 , HDF 3.4 , Kerberos enabled , SSL enabled on Ambari , Ranger , all ranger plugins, ranger kms , HDFS Impact: Ambari shows Total Files + Directories under service metrics as “N/A” Note : This error is shown only for the 3 data nodes Ref: Document followed to enable HDFS SSL - https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-wire-encryption/content/enable_ssl_for_webhdfs_mapreduce_shuffle_tez_and_yarn.html 2019-05-02 09:31:47,711 ERROR [ambari-metrics-retrieval-service-thread-111] MetricsRetrievalService:496 - Unable to retrieve metrics from https://horton-data2.testing.local:45455/jmx. Subsequent failures will be suppressed from the log for 20 minutes. javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching horton-data2.testing.local found at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:142) at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:116) at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:121) at org.apache.ambari.server.state.services.MetricsRetrievalService$MetricRunnable.run(MetricsRetrievalService.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more

Receiving continuous alerts in Nifi-registry log files - Failed to specify server's Kerberos principal name Cluster Specifications: HDP 3.1 HDF 3.4 Kerberos enabled Ranger SSL and PLUGINS enabled for all services All services are green up and running. Only issue is the below log. tail -f nifi-registry-app.log 2019-04-29 10:31:18,219 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Audit Status Log: name=Testcluster_nifi_registry.async.multi_dest.batch.hdfs, interval=01:00.007 minutes, events=1, deferredCount=1, totalEvents=21, totalDeferredCount=21 2019-04-29 10:31:18,219 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Returning HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml 2019-04-29 10:31:18,222 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Checking whether log file exists. hdfPath=hdfs://horton-mgmt1.XXXXXXX.local:8020/ranger/audit/Testcluster_nifi_registry/20190429/Testcluster_nifi_registry_ranger_audit_horton-hdf0.XXXXXXX.local.log, UGI=nifiregistry/horton-hdf0.XXXXXXX.local@XXXXXXX.LOCAL (auth:KERBEROS) 2019-04-29 10:31:18,225 ERROR [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file. java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "horton-hdf0.XXXXXXX.local/xx.xx.xx.xx"; destination host is: "horton-mgmt1.XXXXXXX.local":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0.jar:na] at com.sun.proxy.$Proxy128.getFileInfo(Unknown Source) ~[na:na] at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:874) ~[hadoop-hdfs-client-3.0.0.jar:na] at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0.jar:na] at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source) ~[na:na] at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1697) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1488) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1503) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:289) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) [ranger-plugins-audit-1.1.0.jar:1.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:860) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client.call(Client.java:1383) ~[hadoop-common-3.0.0.jar:na] ... 33 common frames omitted Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:613) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:409) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794) ~[hadoop-common-3.0.0.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794) ~[hadoop-common-3.0.0.jar:na] ... 36 common frames omitted 2019-04-29 10:31:18,225 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.a.destination.HDFSAuditDestination Flushing HDFS audit. Event Size:1 2019-04-29 10:31:18,225 ERROR [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.ranger.audit.queue.AuditFileSpool Error sending logs to consumer. provider=Testcluster_nifi_registry.async.multi_dest.batch, consumer=Testcluster_nifi_registry.async.multi_dest.batch.hdfs 2019-04-29 10:31:18,225 INFO [Testcluster_nifi_registry.async.multi_dest.batch_Testcluster_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.ranger.audit.queue.AuditFileSpool Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=Testcluster_nifi_registry.async.multi_dest.batch, consumer=Testcluster_nifi_registry.async.multi_dest.batch.hdfs NiFi registry configurations : Advanced nifi-registry-authorizers-env <authorizers> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">LDAPS</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Manager DN">CN=xxx,OU=xx,OU=xx,DC=xxx,DC=xxx</property> <property name="Manager Password">xxxxxx</property> <property name="TLS - Keystore">/etc/nifi-registry/conf/certs_stores/nifi-registry-keystore.jks</property> <property name="TLS - Keystore Password"> xxxxxx </property> <property name="TLS - Keystore Type">JKS</property> <property name="TLS - Truststore">/etc/nifi-registry/conf/certs_stores/nifi-registry-truststore.jks</property> <property name="TLS - Truststore Password"> xxxxxx </property> <property name="TLS - Truststore Type">JKS</property> <property name="TLS - Client Auth">WANT</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldaps:// xxxxxx:636</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="User Search Base"></property> <property name="User Object Class">person</property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="User Identity Attribute"></property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">DC= xxxxxx,DC= xxxxxx</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <authorizer> <identifier>ranger-authorizer</identifier> <class>org.apache.nifi.registry.ranger.RangerAuthorizer</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Ranger Audit Config Path">/etc/nifi-registry/conf/ranger-nifi-registry-audit.xml</property> <property name="Ranger Security Config Path">/etc/nifi-registry/conf/ranger-nifi-registry-security.xml</property> <property name="Ranger Service Type">nifi-registry</property> <property name="Ranger Application Id">Testcluster_nifi_registry</property> <property name="Ranger Admin Identity">CN= xxxxxx,OU= xxxxxx,OU= xxxxxx,DC= xxxxxx,DC= xxxxxx </property> <property name="Ranger Kerberos Enabled">true</property> </authorizer> </authorizers> Advanced nifi-registry-identity-providers-env <identityProviders> <provider> <identifier>ldap-identity-provider</identifier> <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class> <property name="Authentication Strategy">LDAPS</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Manager DN">CN=xxxxxx,OU= xxxxxx,OU= xxxxxx,DC= xxxxxx,DC= xxxxxx </property> <property name="Manager Password"> xxxxxx </property> <property name="TLS - Keystore">/etc/nifi-registry/conf/certs_stores/nifi-registry-keystore.jks</property> <property name="TLS - Keystore Password"> xxxxxx </property> <property name="TLS - Keystore Type">JKS</property> <property name="TLS - Truststore">/etc/nifi-registry/conf/certs_stores/nifi-registry-truststore.jks</property> <property name="TLS - Truststore Password"> xxxxxx </property> <property name="TLS - Truststore Type">JKS</property> <property name="TLS - Client Auth">NONE</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldaps:// xxxxxx:636</property> <property name="User Search Base">DC= xxxxxx,DC= xxxxxx </property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> </identityProviders> ... View more

Solr asking for username/password even after providing solr keytab in MIT kerberos on a windows machine C:\Users\Administrator\Desktop>kinit -kt solr.service.keytab solr/horton-mgmt1.h orton.local@HORTON.LOCAL HTTP ERROR 403 Problem accessing /solr/. Reason: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) HTTP ERROR 401 Problem accessing /solr/. Reason: ... View more