Member since
03-22-2019
5
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5286 | 04-03-2019 11:37 PM | |
| 2998 | 03-28-2019 02:04 AM |
04-03-2019
11:37 PM
Hi Gzigldrum, Thank you for the reply, I found the Route cause and resolved the issue myself, my proxy server principal is not listing in the impla demon keytab , impla/proxy@relim in none of the demon nodes, i cross checked the Impala Daemons Load Balancer has confiured to Executor group feild eariler with value " proxyhost:port" this where its not reflecting ,because none of the impla demon nodes are present in executor group . All the impla demon nodes are present in "impala demon default group, i have added the value '"proxyhost:port" in this field and restarted the impala service, when is cross cehck the princaipl in demon node the impala keytab has the impla/proxy@relim, and the kerberos authentication from proxy server started working fro the impala. Regards, Pandu
... View more
03-29-2019
07:15 AM
Hi gzigldurm, Thank you fro the reply, I have review the proxy server and hostname configuration is same ,where i can see the request is coming to proxy and redirect to any onfe of the datanodes where i can see the kerberos ticket and conf file for the proxy and dtanode is mention below proxy server krb5.conf -------------------------- ~]$ cat /etc/krb5.conf [libdefaults] default_realm = DEV.SIT.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 default_tkt_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 permitted_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 udp_preference_limit = 1 kdc_timeout = 3000 [realms] DEV.SIT.COM = { kdc = clouderamanager.hadoop-inventory.local admin_server = clouderamanager.hadoop-inventory.local } [domain_realm] ----------- kerberos Ticket # klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: dbadmin@DEV.SIT.COM Valid starting Expires Service principal 03/29/19 12:00:01 03/30/19 12:00:01 krbtgt/DEV.SIT.COM@DEV.SIT.COM renew until 04/05/19 13:00:01 principals: ------------ $] klist -ket /home/sit/dbadmin.keytab Keytab name: FILE:/home/sit/dbadmin.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 03/18/19 10:38:24 dbadmin@DEV.SIT.COM (arcfour-hmac) ------------------------------------------------------------------------------ the request is redirect to datanode1 and its krb5 and principals ]# cat /etc/krb5.conf [libdefaults] default_realm = DEV.SIT.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 default_tkt_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 permitted_enctypes = rc4-hmac aes256-cts-hmac-sha1-96 udp_preference_limit = 1 kdc_timeout = 3000 [realms] DEV.SIT.COM = { kdc = clouderamanager.hadoop-inventory.local admin_server = clouderamanager.hadoop-inventory.local } [domain_realm] kerberos Ticket in datanode ----------------------------------------------- # klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: impala/data1@DEV.SIT.COM Valid starting Expires Service principal 03/25/19 12:02:59 03/26/19 12:02:59 krbtgt/DEV.SIT.COM@DEV.SIT.COM renew until 04/01/19 13:02:59 principals: ------------------------------------------ ]# klist -ket impala.keytab Keytab name: FILE:impala.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (aes256-cts-hmac-sha1-96) 2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (aes128-cts-hmac-sha1-96) 2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (des3-cbc-sha1) 2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (arcfour-hmac) 2 03/25/19 07:34:08 impala/data1@DEV.SIT.COM (des-hmac-sha1) 2 03/25/19 07:34:08 impala/data1@DEV.SIT.COM (des-cbc-md5) -------------------------------------------------- Implad logs i can see this error E0329 14:04:55.577369 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0329 14:04:55.580916 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0329 14:04:55.672466 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0329 14:04:55.673733 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context from the cloudmanegr the kerberos configuration ]# cat /var/kerberos/krb5kdc/kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] DEV.SIT.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words max_renewable_life = 7d max_life = 1d admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal default_principal_flags = +renewable, +forwardable } where in clouder portal --> administartiom --> settings-- > kerberos kerberos encription types mentions as rc4-hmac aes256-cts-hmac-sha1-96 ------------------------------------------------------ can you please let me know where the issues accoued in the principals? you mean to say rc4-hmac is not supported enscription types Regards, pandu.
... View more
03-28-2019
02:38 AM
Hi All,
We have upgrader he VM resources , like CPU and storage is added to each VM in the Cloudera manager.
we have done this one by one VM , and also we have upgraded the edgeserv as well, where we have proxy is installed to access the datanodes. our appliaction is using the impala api URL to access the data from the datanode via the proxy using edgeserver.
from java side we can see the the error
java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
and checked the edgeserver logs request is recived in proxy and redirct to the data nodes.
in the data nodes logs we can see error logs
E0327 11:57:09.115049 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0327 11:57:09.115571 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0327 11:57:09.199386 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0327 11:57:09.199843 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
i have cheked the kerbros ticked it is valid from both the places. from the appliaction serevr and the edgeserver i am using same .keytab ,file and kerboros ticket is valid.
please help me out to fix this issues, its very urgent to resolve this issue.
Reagrds,
pandu
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager
-
Kerberos
03-28-2019
02:04 AM
Hi, the below issues is resolved, initially the db.properites file is pointing to nav database, thats why the error is osccured, i made changes to point to scm database scheam , then it wokred login into cloudera manager portal. Regards, pandu
... View more
03-22-2019
03:43 AM
Hi All, i tired to start and stop my clodera manger, wheire i need to upgrader the VM with RAM,and storage first i tried to stop the clodera manger #service cloudera-scm-server stop clouderamanger is already stopped but still the UI i was able to access. so i found the process id and kill the process. kill -9 pid and stoped the agent. service cloudera-scm-agent stop and shutdown the VM and allaocted the resourses . and started the VM . now i started the service . service cloudera-scm-server start service cloudera-scm-agent start after startign the service . i see error in the logs 2019-03-22 07:25:40,874 INFO 1894048261@scm-web-0:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:41,317 INFO 1894048261@scm-web-0:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-EVENTSERVER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:41,373 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-REPORTSMANAGER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:41,379 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-HOSTMONITOR-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:41,561 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-SERVICEMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:41,807 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:42,317 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-EVENTSERVER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:42,369 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-REPORTSMANAGER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:42,381 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-HOSTMONITOR-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:42,562 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-SERVICEMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:42,742 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully. 2019-03-22 07:25:42,811 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:42,932 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9659 name=cloudera-mgmt-ALERTPUBLISHER host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,932 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9573 name=yarn-JOBHISTORY host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,932 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=8939 name=zookeeper-server host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,933 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9198 name=hdfs-HTTPFS host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,933 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9199 name=hdfs-SECONDARYNAMENODE host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,933 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9625 name=impala-STATESTORE host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,933 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9657 name=cloudera-mgmt-EVENTSERVER host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,934 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9658 name=cloudera-mgmt-NAVIGATOR host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,934 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9627 name=impala-CATALOGSERVER host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,934 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9660 name=cloudera-mgmt-HOSTMONITOR host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,934 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9661 name=cloudera-mgmt-NAVIGATORMETASERVER host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,934 WARN avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9662 name=cloudera-mgmt-REPORTSMANAGER host=caa40aec-9347-4737-aa2a-e55ae3cabe4f/namenode2 2019-03-22 07:25:42,935 INFO avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=1, hostId=caa40aec-9347-4737-aa2a-e55ae3cabe4f, hostName=namenode2}] Added to rack group: /default 2019-03-22 07:25:43,313 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-EVENTSERVER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:43,371 INFO 59153541@scm-web-3:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-REPORTSMANAGER-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:43,377 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-HOSTMONITOR-bd7c2b94f58360a89cef8b9bbe154e15' from 192.xxx.x.xx 2019-03-22 07:25:43,566 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-SERVICEMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:43,806 INFO 1815833765@scm-web-2:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-06a4fc9d25f0cecc59951cc2023da8a0' from 192.xxx.x.xx 2019-03-22 07:25:43,916 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9536 name=kudu-KUDU_TSERVER host=3b1402e9-c810-48b8-b36d-a2a41dcc0625/data1 2019-03-22 07:25:43,916 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9669 name=yarn-NODEMANAGER host=3b1402e9-c810-48b8-b36d-a2a41dcc0625/data1 2019-03-22 07:25:43,916 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9196 name=hdfs-DATANODE host=3b1402e9-c810-48b8-b36d-a2a41dcc0625/data1 2019-03-22 07:25:43,916 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9684 name=hbase-REGIONSERVER host=3b1402e9-c810-48b8-b36d-a2a41dcc0625/data1 2019-03-22 07:25:43,916 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=9630 name=impala-IMPALAD host=3b1402e9-c810-48b8-b36d-a2a41dcc0625/data1 2019-03-22 07:25:43,917 INFO avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=2, hostId=3b1402e9-c810-48b8-b36d-a2a41dcc0625, hostName=data1}] Added to rack group: /default the same error is occuring repedetialy.. when i login to cloudera manager , it asking for a new setup . please help me to fix this how to go to my cloudera manager UI. and why is shwoing new setup for cloudera manager after restart.
... View more
Labels:
- Labels:
-
Cloudera Manager