Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDSW 1.6.1 Kerberos authentication failing and misleading errors

CDSW 1.6.1 Kerberos authentication failing and misleading errors

Hi Team,

 

I have installed CDSW 1.6.1(latest version available) on my CDH 5.12.2 cluster.

"cdsw status" shows all pods/containers are running without any issues.

But "cdsw validate" throws errors related DNS,ipv6 and [livelog-publisher] pod(s) are not ready.

 

I have successfully tested LDAP authentication. But Kerberos is failing with errors like below:

 

Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.263#0111#011INFO #011AppServer.Models.Users #011Kerberos-Req-3 #011Start Kerberos authentication#011data = {"userId":3,"principal":"p1324642@OPTUS.COM.AU","clusterId":1,"shouldUsePassword":true}
Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011WARNING#011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish Kerberos command, expect failed#011data = {"cmd":"/kinit -c /tmp/tgt-981376665 -f -p myID@DOMAIN.COM","err":"expect: Process not running"}
Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish getting keytab from password, failed initial kinit#011data = {"batchRes":[{"Idx":0,"Output":"kinit: Cannot contact any KDC for realm 'DOMAIN.COM' while getting initial credentials\n","Match":null}],"err":"expect: Process not running","trace":"[25] 1573448639.268159: Getting initial credentials for myID@DOMAIN.COM\n[25] 1573448639.268161: Sending unauthenticated request\n[25] 1573448639.268162: Sending request (186 bytes) to DOMAIN.COM\n[25] 1573448639.268163: Resolving hostname DOMAIN.COM\n"}
Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Server #011aba9a020-0440-11ea-8c0f-bd553e#011Finish testKeytab, failed to get keytab#011data = {"err":"expect: Process not running"}

 

 

cdsw status:

 

Sending detailed logs to [/tmp/cdsw_status_LwaiQM.log] ...
CDSW Version: [1.6.1.1504243:64182a2]
Installed into namespace 'default'
OK: Application running as root check
OK: NFS service check
OK: System process check for CSD install
OK: Sysctl params check
OK: Kernel memory slabs check
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| NAME | STATUS | CREATED-AT | VERSION | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | GPU | STATEFUL |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| <myHostname> | True | 2019-11-11 04:40:59+00:00 | v1.13.9-1+6c8cb1a92335e2 | None | Red Hat Enterprise Linux Server 7.3 (Maipo) | 3.10.0-514.el7.x86_64 | 0 | True |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1/1 nodes are ready.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| etcd-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:10+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
| kube-apiserver-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:29+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
| kube-controller-manager-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:12+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
| kube-dns-86b8794d97-b8djj | 3/3 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.2 | <cdsw_host_IP> | None |
| kube-proxy-68q94 | 1/1 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
| kube-scheduler-<myHostname> | 1/1 | Running | 5 | 2019-11-11 04:41:03+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
| tiller-deploy-64c9844bf6-xd2dr | 1/1 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.3 | <cdsw_host_IP> | None |
| weave-net-rflrn | 2/2 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
All required pods are ready in cluster kube-system.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| cdsw-compute-pod-evaluator-7d6b9b9f8c-bmlhf | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.16 | <cdsw_host_IP> | None |
| cron-555f6b86bd-47tf9 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.4 | <cdsw_host_IP> | cron |
| db-86bbb69b54-skz28 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.11 | <cdsw_host_IP> | db |
| db-migrate-64182a2-57p6h | 0/1 | Succeeded | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.5 | <cdsw_host_IP> | db-migrate |
| ds-cdh-client-75458df965-qhqfj | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.19 | <cdsw_host_IP> | ds-cdh-client |
| ds-operator-55fb686f46-hbl7c | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.25 | <cdsw_host_IP> | ds-operator |
| ds-reconciler-55945b6846-7gpzg | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.20 | <cdsw_host_IP> | ds-reconciler |
| ds-vfs-fbc8544f7-nfmtb | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.17 | <cdsw_host_IP> | ds-vfs |
| image-puller-7scmp | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.23 | <cdsw_host_IP> | image-puller |
| ingress-controller-f98488c49-p8bx2 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.9 | <cdsw_host_IP> | ingress-controller |
| livelog-8c7d64797-psk89 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.10 | <cdsw_host_IP> | livelog |
| livelog-publisher-t2p2c | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.6 | <cdsw_host_IP> | None |
| s2i-builder-6c8f884b9d-868mr | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.22 | <cdsw_host_IP> | s2i-builder |
| s2i-builder-6c8f884b9d-p5j5f | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.21 | <cdsw_host_IP> | s2i-builder |
| s2i-builder-6c8f884b9d-tblz8 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.18 | <cdsw_host_IP> | s2i-builder |
| s2i-client-bf9fdd6f6-vtnm2 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.5 | <cdsw_host_IP> | s2i-client |
| s2i-git-server-888954dbb-cxn7l | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.12 | <cdsw_host_IP> | s2i-git-server |
| s2i-queue-677bdb59f6-dpplg | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.7 | <cdsw_host_IP> | s2i-queue |
| s2i-registry-6d98cd7d87-4mpld | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.14 | <cdsw_host_IP> | s2i-registry |
| s2i-registry-auth-58c6b4885-xmqp8 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.13 | <cdsw_host_IP> | s2i-registry-auth |
| s2i-server-674f958dd8-djx4z | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.15 | <cdsw_host_IP> | s2i-server |
| secret-generator-6b7f85776d-8cl9d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.8 | <cdsw_host_IP> | secret-generator |
| spark-port-forwarder-9bzd9 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | spark-port-forwarder |
| tcp-ingress-controller-7996966bb9-pbx9l | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.24 | <cdsw_host_IP> | tcp-ingress-controller |
| web-85b99b97dc-7dcdc | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.28 | <cdsw_host_IP> | web |
| web-85b99b97dc-j5p8d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.27 | <cdsw_host_IP> | web |
| web-85b99b97dc-kcg27 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.26 | <cdsw_host_IP> | web |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
All required pods are ready in cluster default.
All required Application services are configured.
All required secrets are available.
Persistent volumes are ready.
Persistent volume claims are ready.
Ingresses are ready.
Checking web at url: http://cdswdev.domain.com
OK: HTTP port check
Cloudera Data Science Workbench is ready!

 

 

 

CDSW validate:

 

[root@bdpdevedgla001 bin]# cdsw validate
[Validating host configuration]
> Prechecking OS Version........[OK]
> Prechecking kernel Version........[OK]
> Prechecking that SELinux is disabled........[OK]
> Prechecking scaling limits for processes........[OK]
> Prechecking scaling limits for open files........[OK]
> Loading kernel module [ip_tables]...
> Loading kernel module [iptable_nat]...
> Loading kernel module [iptable_filter]...
> Prechecking that iptables are not configured........[OK]
> Prechecking kernel parameters........[OK]
> Prechecking to ensure kernel memory accounting disabled:........[OK]
> Prechecking Java distribution and version........[OK]
> Checking unlimited Java encryption policy for AES........[OK]
> Prechecking size of root volume........[OK]

[Validating networking setup]
> Checking if kubelet iptables rules exist
> Checking if DNS server is running on localhost
> Checking the number of DNS servers in resolv.conf
> Checking DNS entries for CDSW main domain
> Checking reverse DNS entries for CDSW main domain
WARNING:: DNS doesn't resolve <cdsw_host_IP> to cdswdev.domain.com; DNS is not configured properly: 1
> Checking DNS entries for CDSW wildcard domain
> Checking that firewalld is disabled
> Checking if ipv6 is enabled
WARNING:: ipv6 must be enabled: 1

[Validating Kubernetes versions]
> Checking kubernetes client version
> Checking kubernetes server version

[Validating NFS and Application Block Device setup]
> Checking if nfs or nfs-server is active and enabled
> Checking if rpcbind.socket is active and enabled
> Checking if rpcbind.service is active and enabled
> Checking if the project folder is exported over nfs
> Checking if application mountpoint exists
> Checking if the application directory is on a separate block device
> Checking the root directory (/) free space
> Checking the application directory (/var/lib/cdsw) free space
WARNING:: The directory has less then 10% free capacity: 1

[Validating Kubernetes cluster state]
> Checking if we have exactly one master node
> Checking if the Kubernetes nodes are ready
> Checking kube-apiserver pod
> Checking that kube-system pods are running
> Checking that kube-system pods are ready
> Checking that kube-system services are created
> Checking application pods are running
> Checking that application pods are ready
WARNING: [livelog-publisher] pod(s) are not ready under default namespace.
> Checking that application services are created
> Checking that the application services are reachable
> Checking that the web pods have access to the databases

[Validating CDSW application]
> Checking connectivity over ingress

--------------------------------------------------------------------------
Errors detected.

Please review the issues listed above. Further details can be collected by
capturing logs from all nodes using "cdsw logs".

[root@bdpdevedgla001 bin]#

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here