Authentication Failed in Atlas Hive Import

[#1]

I've already got Atlas working and different hooks configured in other services in my Kerberized CDP Private Cloud Base 7.1.6. I still don't have as many entities as I would expect to see. I tried to sync the Hive Metastore but can't pass an authentication error and cannot find which log to look to get more information.

I've the superuser ticket generated with kinit and when I run the import-hive.sh script in the server where the Hive Metastore is installed, the following message is printed:

[root@<metastore-server> ~]# /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/atlas/hook-bin/import-hive.sh -d inventarios
Using Hive configuration directory [/etc/hive/conf]
/etc/hive/conf:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-yarn/./:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-yarn/.//*
Log file for import is /var/log/atlas/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
10:10:59.450 [main] ERROR org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Import failed
com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:634) ~[jersey-client-1.19.jar:1.19]
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:402) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:366) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:352) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:245) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.AtlasClientV2.getEntityByAttribute(AtlasClientV2.java:376) ~[atlas-client-v2-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.findEntity(HiveMetaStoreBridge.java:820) ~[hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.findDatabase(HiveMetaStoreBridge.java:785) ~[hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:455) ~[hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:323) ~[hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:293) ~[hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:193) [hive-bridge-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
Caused by: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:101) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:165) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 15 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_181]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:237) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:220) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:329) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:99) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:165) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 15 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true&user.name=superuser, status: 500, message: Server Error
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:401) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:74) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) ~[hadoop-auth-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:329) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:99) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) ~[hadoop-common-3.1.1.7.1.6.0-297.jar:?]
at org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(SecureClientUtils.java:94) ~[atlas-client-common-2.1.0.7.1.6.0-297.jar:2.1.0.7.1.6.0-297]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:165) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 15 more
Failed to import Hive Meta Data!!!

I'm still trying to find where to look for more information about the error, since the authentication attempt doesn't appears on the /var/log/atlas/application.log.

Thank you in advance for your help.

[#2]

@wichovalde ,

The error below is a server-side error that should (hopefully) be logged in the Atlas server log.

Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, URL: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=i...
status: 500, message: Server Error

Try to find the entries in the Atlas server log that match this call and it could tell you a bit more about the problem. 

If the Atlas log doesn't seem to have the corresponding information, try setting its log threshold to DEBUG in Cloudera Manager and restarting the service and repeating the test.

André