Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: The Cloudera Community will undergo maintenance on Saturday, August 17 at 12:00am PDT. See more info here.

Impala Catalog Server and Impala Daemons failing to start after enabling Kerberos

Impala Catalog Server and Impala Daemons failing to start after enabling Kerberos

Explorer

My cluster is running Cloudera 5.3, using Cloudera Manager with parcels, the 6 cluster nodes are running CentOS release 6.5.

This post is similair to http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-Catalog-Server-cannot-connect-to...

Metastore-after-enabling/td-p/7606 but has significant differences, namely:

I am using an Active Directory KDC and have followed the documented steps

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_sg_hadoop_security_...

I am using AES-256 Encryption and have installed the policy files.

I have enabled Kerberos via Cloudera Manager and so far zookeeper, hdfs, spark, yarn and hive all start succesfully.

On starting the impala service the statestore starts (running on myhost519) as does an impala daemon on myhost518.  The remaining

impala daemons then fail (running on myhost519, myhost521 and myhost522).  In addition the catalog server fails on myhost519.

CatalogServerLog:
Log file created at: 2015/02/05 08:27:20
Running on machine: myhost519.mydomain.cdh
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0205 08:27:20.832939  5660 logging.cc:102] stdout will be logged to this file.
E0205 08:27:20.833353  5660 logging.cc:103] stderr will be logged to this file.
I0205 08:27:20.838809  5660 authentication.cc:655] Using internal kerberos principal "impala/myhost519.mydomain.cdh@BPATLANT
IS.NET"
I0205 08:27:20.838835  5660 authentication.cc:990] Internal communication is authenticated with Kerberos
I0205 08:27:20.839010  5691 authentication.cc:456] Registering impala/myhost519.mydomain.cdh@mydomain.NET, keytab file /va
r/run/cloudera-scm-agent/process/872-impala-CATALOGSERVER/impala.keytab
I0205 08:27:20.839022  5660 authentication.cc:775] Waiting for Kerberos ticket for principal: impala/myhost519.mydomain.cdh@
mydomain.NET
I0205 08:27:21.295459  5660 authentication.cc:777] Kerberos ticket granted to impala/myhost519.mydomain.cdh@mydomain.NET
I0205 08:27:21.295531  5660 authentication.cc:655] Using external kerberos principal "impala/myhost519.mydomain.cdh@BPATLANT
IS.NET"
I0205 08:27:21.295547  5660 authentication.cc:1006] External communication is authenticated with Kerberos
I0205 08:27:21.295807  5660 init.cc:132] catalogd version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
Built on Tue, 16 Dec 2014 19:25:34 PST
I0205 08:27:21.295819  5660 init.cc:133] Using hostname: myhost519.mydomain.cdh
I0205 08:27:21.296867  5660 logging.cc:138] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=myhost519.mydomain.cdh
--keytab_file=/var/run/cloudera-scm-agent/process/872-impala-CATALOGSERVER/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=80%
--principal=impala/myhost519.mydomain.cdh@mydomain.NET
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=myhost519.mydomain.cdh
--state_store_subscriber_port=23020
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25020
--flagfile=/var/run/cloudera-scm-agent/process/872-impala-CATALOGSERVER/impala-conf/catalogserver_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/catalogd
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0205 08:27:21.296993  5660 init.cc:138] Cpu Info:
  Model: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz
  Cores: 8
  L1 Cache: 24.00 KB
  L2 Cache: 1024.00 KB
  L3 Cache: 0
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
I0205 08:27:21.297008  5660 init.cc:139] Disk Info:
  Num disks 1:
    sda (rotational=true)

I0205 08:27:21.297027  5660 init.cc:140] Physical Memory: 31.34 GB
I0205 08:27:21.297040  5660 init.cc:141] OS version: Linux version 2.6.32-431.17.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org)

(gcc version 4.4.7 20120313 (Red Hat
4.4.7-4) (GCC) ) #1 SMP Wed May 7 23:32:49 UTC 2014
I0205 08:27:21.297049  5660 init.cc:142] Process ID: 5660
I0205 08:27:22.906388  5660 webserver.cc:195] Starting webserver on 0.0.0.0:25020
I0205 08:27:22.906429  5660 webserver.cc:209] Document root: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
I0205 08:27:22.906667  5660 webserver.cc:275] Webserver started
I0205 08:27:23.041636  5660 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0205 08:27:23.043503  5660 JniCatalog.java:89] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_51-b13)
I0205 08:27:23.242524  5691 authentication.cc:484] Successfully renewed Keberos ticket
W0205 08:27:24.269414  5660 HiveConf.java:1507] DEPRECATED: Configuration property hive.metastore.local no longer has any effect.

Make sure to provide a valid value for
 hive.metastore.uris if you are connecting to a remote metastore.
I0205 08:27:24.279480  5660 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0205 08:27:24.299512  5660 HiveMetaStoreClient.java:308] Trying to connect to metastore with URI

thrift://myhost518.mydomain.cdh:9083
E0205 08:27:24.535331  5660 TSaslTransport.java:296] SASL negotiation failure
Java exception follows:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:

Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:347)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:216)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:155)
        at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:56)
        at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:49)
        at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:116)
        at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:93)
        at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:126)
        at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:91)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 18 more
W0205 08:27:24.539211  5660 HiveMetaStoreClient.java:355] Failed to connect to the MetaStore Server...
I0205 08:27:24.539415  5660 HiveMetaStoreClient.java:386] Waiting 1 seconds before next connection attempt.



I have noticed the keytab is not using AES256:





bash-4.1$ klist -e -k -t /var/run/cloudera-scm-agent/process/872-impala-CATALOGSERVER/impala.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/872-impala-CATALOGSERVER/impala.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/04/15 14:42:46 impala/myhost519.mydomain.cdh@mydomain.NET (arcfour-hmac)

The contents of the krb5 file on the hosts:
bash-4.1$ cat /etc/krb5.conf
[libdefaults]
default_realm = mydomain.NET
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
default_tkt_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
permitted_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
udp_preference_limit = 1
[realms]
mydomain.NET = {
kdc = mydc.mydomain.NET
admin_server = mydc.mydomain.NET
}








-------------------------------------------------------------------------------------------
The logs for the failing impala daemons all contain:


Log file created at: 2015/02/05 07:53:50
Running on machine: myhost519.mydomain.cdh
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0205 07:53:50.115967  2765 logging.cc:102] stdout will be logged to this file.
E0205 07:53:50.116345  2765 logging.cc:103] stderr will be logged to this file.
I0205 07:53:50.121875  2765 authentication.cc:655] Using internal kerberos principal "impala/myhost519.mydomain.cdh@mydomain.NET"
I0205 07:53:50.121901  2765 authentication.cc:990] Internal communication is authenticated with Kerberos
I0205 07:53:50.122072  2765 authentication.cc:775] Waiting for Kerberos ticket for principal:

impala/myhost519.mydomain.cdh@mydomain.NET
I0205 07:53:50.122081  2801 authentication.cc:456] Registering impala/myhost519.mydomain.cdh@mydomain.NET, keytab file

/var/run/cloudera-scm-agent/process/869-imp
ala-IMPALAD/impala.keytab
I0205 07:53:50.594002  2765 authentication.cc:777] Kerberos ticket granted to impala/myhost519.mydomain.cdh@mydomain.NET
I0205 07:53:50.594085  2765 authentication.cc:655] Using external kerberos principal "impala/myhost519.mydomain.cdh@mydomain.NET"
I0205 07:53:50.594099  2765 authentication.cc:1006] External communication is authenticated with Kerberos
I0205 07:53:50.594332  2765 init.cc:132] impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
Built on Tue, 16 Dec 2014 19:25:34 PST
I0205 07:53:50.594343  2765 init.cc:133] Using hostname: myhost519.mydomain.cdh
I0205 07:53:50.595392  2765 logging.cc:138] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=myhost519.mydomain.cdh
--keytab_file=/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=11497635840
--principal=impala/myhost519.mydomain.cdh@mydomain.NET
--log_filename=impalad
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=myhost519.mydomain.cdh
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=myhost519.mydomain.cdh
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/var/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/fair-scheduler.xml
--llama_site_path=/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-coINFO0205 07:53:54.271000 main

org.apache.hadoop.yarn.server.resourcemanager.scheduler.
fair.AllocationFileLoaderService] Loading allocation file /var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/fair-

scheduler.xml
INFO0205 07:53:54.468000 AllocationFileReloader

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService] Loading allocation file /var/r
un/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/fair-scheduler.xml
INFO0205 07:53:54.473000 FileWatchThread(/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/llama-site.xml)-0

com.cloudera.impala.util.RequestPoolServic
e] Loading Llama configuration: /var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/llama-site.xml
nf/llama-site.xml
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=false
--audit_event_log_dir=/var/log/impalad/audit
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=false
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--flagfile=/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0205 07:53:50.595512  2765 init.cc:138] Cpu Info:
  Model: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz
  Cores: 8
  L1 Cache: 24.00 KB
  L2 Cache: 1024.00 KB
  L3 Cache: 0
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
I0205 07:53:50.595528  2765 init.cc:139] Disk Info:
  Num disks 1:
    sda (rotational=true)

I0205 07:53:50.595558  2765 init.cc:140] Physical Memory: 31.34 GB
I0205 07:53:50.595572  2765 init.cc:141] OS version: Linux version 2.6.32-431.17.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org)

(gcc version 4.4.7 20120313 (Red Hat
4.4.7-4) (GCC) ) #1 SMP Wed May 7 23:32:49 UTC 2014
I0205 07:53:50.595581  2765 init.cc:142] Process ID: 2765
I0205 07:53:52.568404  2801 authentication.cc:484] Successfully renewed Keberos ticket
I0205 07:53:54.011592  2765 hbase-table-scanner.cc:156] Detected HBase version >= 0.95.2
I0205 07:53:54.492398  2765 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0205 07:53:54.496856  2765 JniFrontend.java:128] Authorization is 'DISABLED'.
I0205 07:53:54.497051  2765 JniFrontend.java:130] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_51-b13)
W0205 07:53:55.410856  2765 HiveConf.java:1507] DEPRECATED: Configuration property hive.metastore.local no longer has any effect.

Make sure to provide a valid value for
 hive.metastore.uris if you are connecting to a remote metastore.
I0205 07:53:55.756598  2765 impala-server.cc:1137] Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 0,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = "",
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
  31: exec_single_node_rows_threshold (i32) = 100,
}
W0205 07:53:57.145812  2765 UserGroupInformation.java:1645] PriviledgedActionException as:impala (auth:KERBEROS)

cause:javax.security.sasl.SaslException: GSS initiate f
ailed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0205 07:53:57.148195  2765 Client.java:674] Exception encountered while connecting to the server :

javax.security.sasl.SaslException: GSS initiate failed [Caused by GS
SException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0205 07:53:57.148593  2765 UserGroupInformation.java:1645] PriviledgedActionException as:impala (auth:KERBEROS)

cause:java.io.IOException: javax.security.sasl.SaslExce
ption: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
I0205 07:53:57.158257  2765 JniFrontend.java:606] java.io.IOException: Failed on local exception: java.io.IOException:

javax.security.sasl.SaslException: GSS initiate f
ailed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details :

local host is: "myhost519.mydomain.
cdh/149.195.61.122"; destination host is: "myhost517.mydomain.cdh":8020;
W0205 07:53:57.204040  2765 UserGroupInformation.java:1645] PriviledgedActionException as:impala (auth:KERBEROS)

cause:javax.security.sasl.SaslException: GSS initiate f
ailed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0205 07:53:57.204872  2765 Client.java:674] Exception encountered while connecting to the server :

javax.security.sasl.SaslException: GSS initiate failed [Caused by GS
SException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0205 07:53:57.205205  2765 UserGroupInformation.java:1645] PriviledgedActionException as:impala (auth:KERBEROS)

cause:java.io.IOException: javax.security.sasl.SaslExce
ption: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0205 07:53:57.205979  2765 JniFrontend.java:534] Cannot detect CDH version. Skipping Hadoop configuration checks
I0205 07:53:57.207105  2765 status.cc:44] Could not read the HDFS root directory at hdfs://myhost517.mydomain.cdh:8020. Error was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No

valid credentials provided (Mechanism
 level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myhost519.mydomain.cdh/149.195.61.122"; destination host

is: "myhost517.mydomain.cdh":
8020;
    @           0x76ce6d  (unknown)
    @           0x94c42d  (unknown)
    @           0x96ce0e  (unknown)
    @           0x96e121  (unknown)
    @           0x74b15c  (unknown)
    @       0x313261ed1d  (unknown)
    @           0x74add9  (unknown)
E0205 07:53:57.207128  2765 impala-server.cc:210] Could not read the HDFS root directory at hdfs://myhost517.mydomain.cdh:8020. Error

was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No

valid credentials provided (Mechanism
 level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myhost519.mydomain.cdh/149.195.61.122"; destination host

is: "myhost517.mydomain.cdh":
8020;
E0205 07:53:57.207206  2765 impala-server.cc:212] Aborting Impala Server startup due to improper configuration

the keytab file contains:
bash-4.1$ klist -e -k -t /var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/869-impala-IMPALAD/impala.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/04/15 14:42:48 impala/myhost519.mydomain.cdh@mydomain.NET (arcfour-hmac)



--------------------------------------------------

for comparison, the log from the one impala daemon that starts succesfully:

Log file created at: 2015/02/05 07:53:19
Running on machine: myhost518.mydomain.cdh
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0205 07:53:19.576716  3418 logging.cc:102] stdout will be logged to this file.
E0205 07:53:19.576985  3418 logging.cc:103] stderr will be logged to this file.
I0205 07:53:19.581945  3418 authentication.cc:655] Using internal kerberos principal "impala/myhost518.mydomain.cdh@mydomain.NET"
I0205 07:53:19.581969  3418 authentication.cc:990] Internal communication is authenticated with Kerberos
I0205 07:53:19.582146  3451 authentication.cc:456] Registering impala/myhost518.mydomain.cdh@mydomain.NET, keytab file

/var/run/cloudera-scm-agent/process/864-imp
ala-IMPALAD/impala.keytab
I0205 07:53:19.582155  3418 authentication.cc:775] Waiting for Kerberos ticket for principal:

impala/myhost518.mydomain.cdh@mydomain.NET
I0205 07:53:19.596633  3418 authentication.cc:777] Kerberos ticket granted to impala/myhost518.mydomain.cdh@mydomain.NET
I0205 07:53:19.596726  3418 authentication.cc:655] Using external kerberos principal "impala/myhost518.mydomain.cdh@mydomain.NET"
I0205 07:53:19.596740  3418 authentication.cc:1006] External communication is authenticated with Kerberos
I0205 07:53:19.596972  3418 init.cc:132] impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
Built on Tue, 16 Dec 2014 19:25:34 PST
I0205 07:53:19.596983  3418 init.cc:133] Using hostname: myhost518.mydomain.cdh
I0205 07:53:19.598024  3418 logging.cc:138] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=myhost518.mydomain.cdh
--keytab_file=/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=11076108288
--principal=impala/myhost518.mydomain.cdh@mydomain.NET
--log_filename=impalad
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=myhost519.mydomain.cdh
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=myhost519.mydomain.cdh
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/var/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-conf/fair-scheduler.xml
--llama_site_path=/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-coINFO0205 07:53:23.150000 main

org.apache.hadoop.yarn.server.resourcemanager.scheduler.
fair.AllocationFileLoaderService] Loading allocation file /var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-conf/fair-

scheduler.xml
INFO0205 07:53:23.338000 FileWatchThread(/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-conf/llama-site.xml)-0

com.cloudera.impala.util.RequestPoolServic
e] Loading Llama configuration: /var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-conf/llama-site.xml
nf/llama-site.xml
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=false
--audit_event_log_dir=/var/log/impalad/audit
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=false
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--flagfile=/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0205 07:53:19.598145  3418 init.cc:138] Cpu Info:
  Model: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz
  Cores: 8
  L1 Cache: 24.00 KB
  L2 Cache: 1024.00 KB
  L3 Cache: 0
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
I0205 07:53:19.598160  3418 init.cc:139] Disk Info:
  Num disks 1:
    sda (rotational=true)

I0205 07:53:19.598179  3418 init.cc:140] Physical Memory: 31.34 GB
I0205 07:53:19.598192  3418 init.cc:141] OS version: Linux version 2.6.32-431.17.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org)

(gcc version 4.4.7 20120313 (Red Hat
4.4.7-4) (GCC) ) #1 SMP Wed May 7 23:32:49 UTC 2014
I0205 07:53:19.598201  3418 init.cc:142] Process ID: 3418
I0205 07:53:21.109642  3451 authentication.cc:484] Successfully renewed Keberos ticket
I0205 07:53:22.908138  3418 hbase-table-scanner.cc:156] Detected HBase version >= 0.95.2
I0205 07:53:23.357183  3418 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0205 07:53:23.361599  3418 JniFrontend.java:128] Authorization is 'DISABLED'.
I0205 07:53:23.361811  3418 JniFrontend.java:130] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_51-b13)
W0205 07:53:24.086340  3418 HiveConf.java:1507] DEPRECATED: Configuration property hive.metastore.local no longer has any effect.

Make sure to provide a valid value for
 hive.metastore.uris if you are connecting to a remote metastore.
I0205 07:53:24.430191  3418 impala-server.cc:1137] Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 0,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = "",
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
  31: exec_single_node_rows_threshold (i32) = 100,
}
I0205 07:53:26.567344  3418 tmp-file-mgr.cc:77] Created the following scratch dirs:/tmp/impala-scratch
I0205 07:53:26.567513  3418 simple-logger.cc:72] Logging to: /var/log/impalad/profiles//impala_profile_log_1.0-1423122806567
I0205 07:53:26.567812  3418 simple-logger.cc:72] Logging to: /var/log/impalad/audit/impala_audit_event_log_1.0-1423122806567
I0205 07:53:26.602108  3418 impala-server.cc:1857] Impala Beeswax Service listening on 21000
I0205 07:53:26.603585  3418 impala-server.cc:1879] Impala HiveServer2 Service listening on 21050
I0205 07:53:26.605016  3418 impala-server.cc:1894] ImpalaInternalService listening on 22000
I0205 07:53:26.605340  3418 thrift-server.cc:389] ThriftServer 'backend' started on port: 22000
I0205 07:53:26.605357  3418 exec-env.cc:277] Starting global services
I0205 07:53:26.631554  3418 exec-env.cc:338] Using global memory limit: 10.32 GB
I0205 07:53:26.631806  3418 webserver.cc:195] Starting webserver on 0.0.0.0:25000
I0205 07:53:26.631824  3418 webserver.cc:209] Document root: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
I0205 07:53:26.632050  3418 webserver.cc:275] Webserver started
I0205 07:53:26.632072  3418 simple-scheduler.cc:181] Starting simple scheduler
I0205 07:53:26.632362  3418 simple-scheduler.cc:232] Simple-scheduler using 149.195.61.121 as IP address
I0205 07:53:26.632391  3418 statestore-subscriber.cc:186] Starting statestore subscriber
I0205 07:53:26.634208  3418 thrift-server.cc:389] ThriftServer 'StatestoreSubscriber' started on port: 23000
I0205 07:53:26.634229  3418 statestore-subscriber.cc:197] Registering with statestore
I0205 07:53:26.691769  3418 statestore-subscriber.cc:170] Subscriber registration ID: 8942a5574993af30:6cb68f6b1469e38d
I0205 07:53:26.691793  3418 statestore-subscriber.cc:201] statestore registration successful
I0205 07:53:26.697487  3418 thrift-server.cc:389] ThriftServer 'beeswax-frontend' started on port: 21000
I0205 07:53:26.703851  3418 thrift-server.cc:389] ThriftServer 'hiveserver2-frontend' started on port: 21050
I0205 07:53:26.703876  3418 impalad-main.cc:89] Impala has started.
I0205 07:53:26.826583  3522 authentication.cc:383] Successfully authenticated principal "impala/myhost519.mydomain.cdh@mydomain.NET"

on an internal connection
I0205 07:53:26.859779  3522 authentication.cc:383] Successfully authenticated principal "impala/myhost519.mydomain.cdh@mydomain.NET"

on an internal connection
I0205 07:53:26.860493  3657 simple-scheduler.cc:341] Registering local backend with statestore


the keytab
bash-4.1$ klist -e -k -t /var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/864-impala-IMPALAD/impala.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/04/15 14:42:47 impala/myhost518.mydomain.cdh@mydomain.NET (arcfour-hmac)



and the krb conf file:

bash-4.1$ cat /etc/krb5.conf
[libdefaults]
default_realm = mydomain.NET
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
default_tkt_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
permitted_enctypes = aes256-cts-hmac-sha1-96 arcfour-hmac rc4-hmac
udp_preference_limit = 1
[realms]
mydomain.NET = {
kdc = mydc.mydomain.NET
admin_server = mydc.mydomain.NET
}

--------------------------

re: the catalogue server failing:
It looks like the catalogue server is failing to talk to the hive metastore over thrift - has anyone a suggestion of how to fix this?

re: the failing impala daemons - I am stuck on this, I thought it was related to the impala daemons not using AES256, hoewver as one

of the daemon is starting and not using AES256 then it can't be because of this.  does anyone have any suggestions?

thanks
Pete

2 REPLIES 2

Re: Impala Catalog Server and Impala Daemons failing to start after enabling Kerberos

Explorer

Did you able to fix it? I am having same issue. If you could post the solution it will be helpful. 

Re: Impala Catalog Server and Impala Daemons failing to start after enabling Kerberos

New Contributor

I had similar problem after upgrading from 5.13 to C6.2. In my case I had to remove below settings in order to get Impala to started.

 

configs conflicting,

--ssl_minimum_version=tlsv1.2

 

log errors:

I0627 23:41:36.248548 61768 webserver.cc:361] Webserver: Unsupported TLS version: tlsv1.2
I0627 23:41:36.252027 61768 status.cc:122] Webserver: Could not start on address 0.0.0.0:25000
@ 0x83edf9 impala::Status::Status()
@ 0xbdd908 impala::Webserver::Start()
@ 0x9dc304 impala::ExecEnv::StartServices()
@ 0xb1a6cb ImpaladMain()
@ 0x7d99b3 main
@ 0x7f090cdc3d20 __libc_start_main
@ 0x80c94d (unknown)
E0627 23:41:36.252037 61768 impalad-main.cc:95] Impalad services did not start correctly, exiting. Error: Webserver: Could not start on address 0.0.0.0:25000

 

 

Thanks,

Narendra Jonna