Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Catalog Server cannot connect to Metastore after enabling Kerberos

Highlighted

Impala Catalog Server cannot connect to Metastore after enabling Kerberos

Explorer

I have a CDH5 Beta 2 cluster (2 NN, 5 DN, 1 Mgmt Node), and am using Cloudera Manager with parcels.  All operating systems are RHEL 6.5. 

 

I've enabled Kerberos security in Cloudera Manager, and begun restarting services.  So far, zookeeper, hdfs, YARN and Hive services have all started correctly with Kerberos enabled.

 

When attempting to start Impala, the Catalog Server Daemon fails when attempting to connect to the Hive MetaStore service.  The error string seems to indicate that it has failed to obtain a Kerberos tgt (ticket) from the KDC ... yet the first 5 lines in the log file indicate that a ticket has been successfully granted. I've posted the output from the INFO log and supporting info at the bottom of this post.

 

I found a similar thread on these forums, which suggested upping the renewlife to 7days and +allow_renewable on the impala princ, but those variables are already set (see the bottom of this post).

 

Any suggestions on what to look at next?

 

Thanks,

James

 

 

Error in /var/log/catalogd/catalogd.INFO:

I0319 17:49:14.094182  4537 authentication.cc:399] Waiting for Kerberos ticket for principal: impala/server-svb-0020@UKHADOOP
I0319 17:49:14.094178  4569 authentication.cc:275] Registering impala/server-svb-0020@UKHADOOP key_tab file /var/run/cloudera-scm-agent/process/1083-impala-CATALOGSERVER/impala.keytab
I0319 17:49:14.109714  4569 authentication.cc:311] kinit returned: ''
I0319 17:49:14.109799  4537 authentication.cc:401] Kerberos ticket granted to impala/server-svb-0020@UKHADOOP
I0319 17:49:14.110043  4537 init.cc:72] catalogd version 1.2.3-cdh5.0.0-beta-2 RELEASE (build 8e266e052e423af592871e2dfe09d54c03f6a0e8)
Built on Fri, 07 Feb 2014 11:40:43 PST
I0319 17:49:14.110052  4537 init.cc:73] Using hostname: server-svb-0020
I0319 17:49:14.110589  4537 logging.cc:100] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--dump_ir=false
--module_output=
--abort_on_config_error=true
--be_port=22000
--be_principal=impala/server-svb-0020@UKHADOOP
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=server-svb-0020
--keytab_file=/var/run/cloudera-scm-agent/process/1083-impala-CATALOGSERVER/impala.keytab
--mem_limit=80%
--principal=impala/server-svb-0020@UKHADOOP
--log_filename=catalogd
--exchg_node_buffer_size_bytes=10485760
--max_row_batches=0
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_manual_config=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--reuse_io_buffers=true
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_callback_port=28000
--llama_host=127.0.0.1
--llama_port=15000
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=10
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=server-svb-0020
--state_store_subscriber_port=23020
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--log_mem_usage_interval=0
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--pool_conf_file=
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=10
--state_store_port=24000
--statestore_heartbeat_frequency_ms=500
--statestore_max_missed_heartbeats=5
--statestore_num_heartbeat_threads=10
--statestore_suspect_heartbeats=2
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.0.0-0.cdh5b2.p0.27/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25020
--flagfile=/var/run/cloudera-scm-agent/process/1083-impala-CATALOGSERVER/impala-conf/catalogserver_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/catalogd
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=2
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
.
.
.
I0319 17:49:15.128918  4537 MetaStoreClientPool.java:46] Creating MetaStoreClient. Pool Size = 0
I0319 17:49:15.149142  4537 HiveMetaStoreClient.java:249] Trying to connect to metastore with URI thrift://server-svb-0020:9083
E0319 17:49:15.208865  4537 TSaslTransport.java:296] SASL negotiation failure
Java exception follows:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:288)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:109)
        at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:47)
        at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:40)
        at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:105)
        at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:112)
        at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:59)
        at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:50)
        at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:77)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

 

 

 

All of the principals are successfully created for the services in the KDC:

[root@server-svb-0020 ~]# kadmin.local -q "getprincs"
HTTP/server-hdp-0001@UKHADOOP
HTTP/server-hdp-0002@UKHADOOP
HTTP/server-hdp-0003@UKHADOOP
HTTP/server-hdp-0004@UKHADOOP
HTTP/server-hdp-0005@UKHADOOP
HTTP/server-svb-0018@UKHADOOP
HTTP/server-svb-0019@UKHADOOP
HTTP/server-svb-0020@UKHADOOP
K/M@UKHADOOP
cloudera-scm/admin@UKHADOOP
hdfs/server-hdp-0001@UKHADOOP
hdfs/server-hdp-0002@UKHADOOP
hdfs/server-hdp-0003@UKHADOOP
hdfs/server-hdp-0004@UKHADOOP
hdfs/server-hdp-0005@UKHADOOP
hdfs/server-svb-0018@UKHADOOP
hdfs/server-svb-0019@UKHADOOP
hdfs/server-svb-0020@UKHADOOP
hive/server-hdp-0001@UKHADOOP
hive/server-hdp-0002@UKHADOOP
hive/server-hdp-0003@UKHADOOP
hive/server-hdp-0004@UKHADOOP
hive/server-hdp-0005@UKHADOOP
hive/server-svb-0020@UKHADOOP
httpfs/server-hdp-0001@UKHADOOP
httpfs/server-hdp-0002@UKHADOOP
httpfs/server-hdp-0003@UKHADOOP
httpfs/server-hdp-0004@UKHADOOP
httpfs/server-hdp-0005@UKHADOOP
httpfs/server-svb-0018@UKHADOOP
httpfs/server-svb-0019@UKHADOOP
httpfs/server-svb-0020@UKHADOOP
hue/server-svb-0020@UKHADOOP
impala/server-hdp-0001@UKHADOOP
impala/server-hdp-0002@UKHADOOP
impala/server-hdp-0003@UKHADOOP
impala/server-hdp-0004@UKHADOOP
impala/server-hdp-0005@UKHADOOP
impala/server-svb-0020@UKHADOOP
jcadmin/admin@UKHADOOP
kadmin/admin@UKHADOOP
kadmin/changepw@UKHADOOP
kadmin/server-svb-0020@UKHADOOP
krbtgt/UKHADOOP@UKHADOOP
mapred/server-svb-0020@UKHADOOP
oozie/server-svb-0020@UKHADOOP
yarn/server-hdp-0001@UKHADOOP
yarn/server-hdp-0002@UKHADOOP
yarn/server-hdp-0003@UKHADOOP
yarn/server-hdp-0004@UKHADOOP
yarn/server-hdp-0005@UKHADOOP
yarn/server-svb-0020@UKHADOOP

 

The keytab has both the HTTP and impala keys:

[root@server-svb-0020 ~]# klist -e -k -t /var/run/cloudera-scm-agent/process/1083-impala-CATALOGSERVER/impala.keytab

Keytab name: FILE:/var/run/cloudera-scm-agent/process/1083-impala-CATALOGSERVER/impala.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   5 03/19/14 13:43:24 impala/server-svb-0020@UKHADOOP (aes128-cts-hmac-sha1-96) 
   5 03/19/14 13:43:24 impala/server-svb-0020@UKHADOOP (des3-cbc-sha1) 
   5 03/19/14 13:43:24 impala/server-svb-0020@UKHADOOP (arcfour-hmac) 
   5 03/19/14 13:43:24 impala/server-svb-0020@UKHADOOP (des-hmac-sha1) 
   5 03/19/14 13:43:24 impala/server-svb-0020@UKHADOOP (des-cbc-md5) 
   6 03/19/14 13:43:24 HTTP/server-svb-0020@UKHADOOP (aes128-cts-hmac-sha1-96) 
   6 03/19/14 13:43:24 HTTP/server-svb-0020@UKHADOOP (des3-cbc-sha1) 
   6 03/19/14 13:43:24 HTTP/server-svb-0020@UKHADOOP (arcfour-hmac) 
   6 03/19/14 13:43:24 HTTP/server-svb-0020@UKHADOOP (des-hmac-sha1) 
   6 03/19/14 13:43:24 HTTP/server-svb-0020@UKHADOOP (des-cbc-md5) 

 

 

The principal information for impala:

[root@server-svb-0020 ~]# kadmin.local -q "getprinc impala/server-svb-0020@UKHADOOP"

Authenticating as principal jcadmin/admin@UKHADOOP with password.
Principal: impala/server-svb-0020@UKHADOOP
Expiration date: [never]
Last password change: Wed Mar 19 13:43:24 GMT 2014
Password expiration date: [none]
Maximum ticket life: 14 days 00:00:00
Maximum renewable life: 365 days 00:00:00
Last modified: Wed Mar 19 14:47:46 GMT 2014 (jcadmin/admin@UKHADOOP)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 5
Key: vno 5, aes128-cts-hmac-sha1-96, no salt
Key: vno 5, des3-cbc-sha1, no salt
Key: vno 5, arcfour-hmac, no salt
Key: vno 5, des-hmac-sha1, no salt
Key: vno 5, des-cbc-md5, no salt
MKey: vno 1
Attributes:
Policy: [none]

 

 

The imapla user has a ticket:

[impala@server-svb-0020 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_483
Default principal: impala/server-svb-0020@UKHADOOP

Valid starting     Expires            Service principal
03/19/14 18:42:23  04/02/14 19:42:23  krbtgt/UKHADOOP@UKHADOOP
        renew until 04/02/14 19:42:23

 

1 REPLY 1

Re: Impala Catalog Server cannot connect to Metastore after enabling Kerberos

Explorer

Resolved.

 

It looks like the krbtgt principal had the aes256 key, though no other principals did.  Apparently, the impala principal was somehow pulling tickets that had the aes256 encryption type. I didn't want to bother with the JCE policy stuff on a proof of concept cluster, so I rebuilt the KDC with just aes128 and now everything is working.

 

James