Support Questions
Find answers, ask questions, and share your expertise

Problem with impala after tls\ssl enabling

Rising Star

Version: Cloudera Enterprise Data Hub Edition Trial 5.8.3 

Java VM Vendor: Oracle Corporation

Java Version: 1.8.0_60

CDH: 5.7.5

OS RHEL: 7.2

 

Hi Guys, 

I have o problem with running Impala with TLS\SSL encryption. Certificates are signed and went through SSL verification. 

openssl verify cert.pem
cert.pem: OK

Here is the impala log:

Dec 19, 5:48:35.345 PM  INFO  logging.cc:119  
stdout will be logged to this file.
Dec 19, 5:48:35.345 PM  ERROR logging.cc:120  
stderr will be logged to this file.
Dec 19, 5:48:35.346 PM  INFO  authentication.cc:1016  
Internal communication is not authenticated
Dec 19, 5:48:35.346 PM  INFO  authentication.cc:1037  
External communication is not authenticated
Dec 19, 5:48:35.346 PM  INFO  init.cc:159 
impalad version 2.5.0-cdh5.7.5 RELEASE (build ed04e8c47ab0711f818386b471683ee3e5f403a9)
Built on Wed, 02 Nov 2016 12:05:43 PST
Dec 19, 5:48:35.346 PM  INFO  init.cc:160 
Using hostname: ukgs2hdn03.cwglobal.local
Dec 19, 5:48:35.346 PM  INFO  logging.cc:155  
Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_accept_queue_server=true
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=ukgs2hdn03.cwglobal.local
--keytab_file=
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--mem_limit=53687091200
--principal=
--redaction_rules_file=
--max_log_files=10
--log_filename=impalad
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--enable_phj_probe_side_filtering=true
--accepted_cnxn_queue_depth=10000
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=ukgs2hmn02.cwglobal.local
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=ukgs2hmn02.cwglobal.local
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/var/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/data0/impala/impalad,/data1/impala/impalad,/data2/impala/impalad,/data3/impala/impalad,/data4/impala/impalad,/data5/impala/impalad,/data6/impala/impalad,/data7/impala/impalad,/data8/impala/impalad,/data9/impala/impalad
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=true
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=false
--abort_on_failed_lineage_event=true
--audit_event_log_dir=/var/log/impalad/audit
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=/var/log/impalad/lineage
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=/opt/certificates/x509/truststore.pem
--ssl_private_key=/opt/certificates/x509/key.pem
--ssl_private_key_password_cmd=
--ssl_server_certificate=/opt/certificates/x509/cert.pem
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=/opt/certificates/x509/cert.pem
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.7.5-1.cdh5.7.5.p0.3/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--webserver_private_key_file=/opt/certificates/x509/key.pem
--webserver_private_key_password_cmd=
--flagfile=/run/cloudera-scm-agent/process/495-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
Dec 19, 5:48:35.346 PM  INFO  init.cc:165 
Cpu Info:
  Model: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
  Cores: 64
  L1 Cache: 32.00 KB (Line: 64.00 B)
  L1 Cache: 256.00 KB (Line: 64.00 B)
  L1 Cache: 40.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
Dec 19, 5:48:35.346 PM  INFO  init.cc:166 
Disk Info: 
  Num disks 11: 
    sdb (rotational=true)
    sda (rotational=true)
    sdc (rotational=true)
    sdd (rotational=true)
    sde (rotational=true)
    sdf (rotational=true)
    sdg (rotational=true)
    sdh (rotational=true)
    sdi (rotational=true)
    sdj (rotational=true)
    sdk (rotational=true)

Dec 19, 5:48:35.346 PM  INFO  init.cc:167 
Physical Memory: 503.79 GB
Dec 19, 5:48:35.346 PM  INFO  init.cc:168 
OS version: Linux version 3.10.0-327.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
Dec 19, 5:48:35.346 PM  INFO  init.cc:169 
Process ID: 49111
Dec 19, 5:48:36.436 PM  INFO  hbase-table-scanner.cc:157  
Detected HBase version >= 0.95.2
Dec 19, 5:48:36.450 PM  INFO  GlogAppender.java:123 
Logging initialized. Impala: VLOG, All other: INFO
Dec 19, 5:48:36.452 PM  INFO  JniFrontend.java:129  
Authorization is 'DISABLED'.
Dec 19, 5:48:36.452 PM  INFO  JniFrontend.java:131  
Java Version Info: Java(TM) SE Runtime Environment (1.8.0_60-b27)
Dec 19, 5:48:36.733 PM  INFO  simple-scheduler.cc:90  
Admission control is disabled.
Dec 19, 5:48:36.734 PM  INFO  impala-server.cc:1148 
Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 0,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = "",
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
  31: exec_single_node_rows_threshold (i32) = 100,
  32: optimize_partition_key_scans (bool) = false,
  34: schedule_random_replica (bool) = false,
  35: scan_node_codegen_threshold (i64) = 1800000,
  36: disable_streaming_preaggregations (bool) = false,
  37: runtime_filter_mode (i32) = 1,
  38: runtime_bloom_filter_size (i32) = 1048576,
  39: runtime_filter_wait_time_ms (i32) = 0,
  40: disable_row_runtime_filtering (bool) = false,
  41: max_num_runtime_filters (i32) = 10,
}
Dec 19, 5:48:36.897 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data0/impala/impalad/impala-scratch on disk 0
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data1/impala/impalad/impala-scratch on disk 2
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data2/impala/impalad/impala-scratch on disk 3
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data3/impala/impalad/impala-scratch on disk 4
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data4/impala/impalad/impala-scratch on disk 5
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data5/impala/impalad/impala-scratch on disk 6
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data6/impala/impalad/impala-scratch on disk 7
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data7/impala/impalad/impala-scratch on disk 8
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data8/impala/impalad/impala-scratch on disk 9
Dec 19, 5:48:36.898 PM  INFO  tmp-file-mgr.cc:106 
Using scratch directory /data9/impala/impalad/impala-scratch on disk 10
Dec 19, 5:48:36.898 PM  INFO  simple-logger.cc:76 
Logging to: /var/log/impalad/profiles//impala_profile_log_1.1-1482169716898
Dec 19, 5:48:36.898 PM  INFO  simple-logger.cc:76 
Logging to: /var/log/impalad/audit/impala_audit_event_log_1.0-1482169716898
Dec 19, 5:48:36.898 PM  INFO  simple-logger.cc:76 
Logging to: /var/log/impalad/lineage/impala_lineage_log_1.0-1482169716898
Dec 19, 5:48:36.912 PM  INFO  impala-server.cc:1849 
Enabling SSL for Beeswax
Dec 19, 5:48:36.912 PM  INFO  impala-server.cc:1854 
Impala Beeswax Service listening on 21000
Dec 19, 5:48:36.913 PM  INFO  impala-server.cc:1871 
Enabling SSL for HiveServer2
Dec 19, 5:48:36.913 PM  INFO  impala-server.cc:1876 
Impala HiveServer2 Service listening on 21050
Dec 19, 5:48:36.914 PM  INFO  impala-server.cc:1891 
Enabling SSL for backend
Dec 19, 5:48:36.914 PM  INFO  impala-server.cc:1896 
ImpalaInternalService listening on 22000
Dec 19, 5:48:36.916 PM  INFO  thrift-server.cc:446  
ThriftServer 'backend' started on port: 22000s
Dec 19, 5:48:36.916 PM  INFO  exec-env.cc:330 
Starting global services
Dec 19, 5:48:36.924 PM  INFO  exec-env.cc:417 
Using global memory limit: 50.00 GB
Dec 19, 5:48:36.925 PM  INFO  webserver.cc:216  
Starting webserver on 0.0.0.0:25000
Dec 19, 5:48:36.925 PM  INFO  webserver.cc:222  
Webserver: Enabling HTTPS support
Dec 19, 5:48:36.925 PM  INFO  webserver.cc:230  
Document root: /opt/cloudera/parcels/CDH-5.7.5-1.cdh5.7.5.p0.3/lib/impala
Dec 19, 5:48:36.926 PM  INFO  webserver.cc:315  
Webserver started
Dec 19, 5:48:36.926 PM  INFO  simple-scheduler.cc:170 
Starting simple scheduler
Dec 19, 5:48:36.926 PM  INFO  simple-scheduler.cc:218 
Simple-scheduler using 10.0.1.206 as IP address
Dec 19, 5:48:36.926 PM  INFO  statestore-subscriber.cc:179  
Starting statestore subscriber
Dec 19, 5:48:36.927 PM  INFO  thrift-server.cc:446  
ThriftServer 'StatestoreSubscriber' started on port: 23000
Dec 19, 5:48:36.927 PM  INFO  statestore-subscriber.cc:190  
Registering with statestore
Dec 19, 5:48:36.941 PM  INFO  client-cache.h:259  
client 0xe97be00 unexpected exception: SSL_get_verify_result(), unable to get issuer certificate, type=N6apache6thrift9transport13TSSLExceptionE
Dec 19, 5:48:36.941 PM  INFO  client-cache.cc:80  
ReopenClient(): re-creating client for ukgs2hmn02.cwglobal.local:24000
Dec 19, 5:48:36.954 PM  INFO  status.cc:45  
RPC Error: SSL_get_verify_result(), unable to get issuer certificate
    @           0x7b1aba  (unknown)
    @           0xa8ffdb  (unknown)
    @           0xa8a846  (unknown)
    @           0xa8c2a0  (unknown)
    @           0x9541ce  (unknown)
    @           0xa6d9d2  (unknown)
    @           0x75b933  (unknown)
    @     0x7efd97d23b35  __libc_start_main
    @           0x77e6ad  (unknown)
Dec 19, 5:48:36.954 PM  INFO  client-cache.cc:158 
Broken Connection, destroy client for ukgs2hmn02.cwglobal.local:24000
Dec 19, 5:48:36.955 PM  INFO  statestore-subscriber.cc:196  
statestore registration unsuccessful: RPC Error: SSL_get_verify_result(), unable to get issuer certificate
Dec 19, 5:48:36.955 PM  ERROR impalad-main.cc:82  
Impalad services did not start correctly, exiting.  Error: RPC Error: SSL_get_verify_result(), unable to get issuer certificate
Statestore subscriber did not start up.

 

All other services work fine with these certs.

Thanks,

Andrzej

 

1 REPLY 1

Re: Problem with impala after tls\ssl enabling

New Contributor
how did you sort this issue with ssl?