not ready yet: some system pods are not ready

[#1]

Hi

the statas command shows "cdsw status" is 

Cloudera Data Science Workbench is not ready yet: some system pods are not ready

the Logs commands shows" cdsw logs"
Generating Cloudera Data Science Workbench diagnostic bundle...
Checking system basics...
Saving kernel parameters...
Checking validation output...
Checking application configuration...
Checking disks...
Checking Hadoop configuration...
Checking network...
Checking system services...
Checking Docker...
Checking Kubernetes...
Checking Kubelet...
Checking application services...
Checking cluster info...
Checking app cluster info...
Exporting user ids...
ERROR: relation "users" does not exist
LINE 1: SELECT username_hash as uid from users where deactivated='fa...

please assist guys

Regards

S

[#2]

Hi NES,

Could you share the full 'cdsw status' output? I would like to see what system pods are not ready.

There is a db-migrate-XXXXX-xxxxx Kubernetes job which should make sure that the tables are created. When you submit the 'cdsw status' command, can you see this job in the output? Did you see a Completed status for this?

When you changed the cdsw configuration, did you submit a 'cdsw reset' before retrying the 'cdsw init'? If not, you will need to do a 'cdsw reset' and 'cdsw init' again.

'cdsw logs' generates a tar.gz file with different diagnostic information. If you want to validate your cluster, I recommend to use the 'cdsw validate' instead.

Thanks,

Peter

[#3]

hi 

thanks for assisting Peter

here is the output. i have done the cdsw reset and then init.

cdsw status
Cloudera Data Science Workbench Status

Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...

Node Status
NAME STATUS AGE STATEFUL
master.local Ready 28m true

System Pod status
NAME READY STATUS RESTARTS AGE IP NODE
dummy-2088944543-90otk 1/1 Running 0 27m 10.x.x.x master.local
etcd-master.local 1/1 Running 0 28m 10.x.x.x master.local
kube-apiserver-master.local 1/1 Running 0 28m 10.x.x.x master.local
kube-controller-manager-master.local 1/1 Running 0 28m 10.x.x.x master.local
kube-discovery-1150918428-p4bcn 1/1 Running 0 27m 10.x.x.x master.local
kube-dns-3873593988-gxvyr 3/3 Running 0 26m 100.66.0.2 master.local
kube-proxy-bru51 1/1 Running 0 27m 10.x.x.x master.local
kube-scheduler-master.local 1/1 Running 0 28m 10.x.x.x master.local
node-problem-detector-v0.1-xgfcd 1/1 Running 0 26m 10.x.x.x master.local
weave-net-p0p53 2/2 Running 0 27m 10.x.x.x master.local

Cloudera Data Science Workbench Pod Status
NAME READY STATUS RESTARTS AGE IP NODE ROLE
cron-2934152315-keyj7 1/1 Running 0 26m 100.66.0.7 master.local cron
db-39862959-9aw8e 1/1 Running 0 26m 100.66.0.4 master.local db
db-migrate-052787a-wgkjq 0/1 Completed 0 26m 100.66.0.5 master.local db-migrate
engine-deps-m39yd 1/1 Running 0 26m 100.66.0.3 master.local engine-deps
ingress-controller-3138093376-2gct2 1/1 Running 0 26m 10.x.x.x master.local ingress-controller
livelog-1900214889-0gi1v 1/1 Running 0 26m 100.66.0.6 master.local livelog
reconciler-459456250-qnuzc 1/1 Running 0 26m 100.66.0.8 master.local reconciler
spark-port-forwarder-dxhjm 1/1 Running 0 26m 10.x.x.x master.local spark-port-forwarder
web-3826671331-5b7wk 0/1 ContainerCreating 0 26m <none> master.local web
web-3826671331-63il0 0/1 ContainerCreating 0 26m <none> master.local web
web-3826671331-hf86v 0/1 ContainerCreating 0 26m <none> master.local web

Cloudera Data Science Workbench is not ready yet: some application pods are not ready
[root@master ~]# cdsw validate
Checking services...
Checking if docker is active and enabled
Checking if docker is responsive
Checking if kubelet is active and enabled
Check if docker monitor is active
Testing networking setup...
Check if kubelet iptables rules exist
Check that firewalld is disabled
Check configuration file...
Checking master node filesystem configuration...
Checking kubernetes
Checking system pods
Checking application pods exist
Checking application pods are running

ERROR:: Application pod web-3826671331-5b7wk is not running, its state is ContainerCreating: 1

[#4]

Hi NES,

I see that the web pods are stuck in ContainerCreating state. 

I would try to submit a kubectl describe command on one of the web pods and check if we have any associated events. 

$ kubectl describe pod web-3826671331-5b7wk

If there are no events I would check the kubelet logs, maybe it shows the reason why are we stuck:

# get the kubelet logs in less

$ journalctl -u kubelet

# tail the kubelet logs

$ journalctl -u kubelet -o cat -l -f

Regards,

Peter

[#5]

hi

thanks, you are the best Peter.

i managed to find the issue via command " kubectl describe pod web-3826671331-5b7wk" and i am playing using it now.

Regard

Nes

[#6]

Hi NES,

I'm glad that your issue is resolved!

Could you let us know what was the issue for the other topic followers?

Thanks,

Peter

[#7 · Accepted solution]

Hi

the problem when stuck in ContainerCreating state, base on logs generated by command"kubectl describe pod web-3826671331-5b7wk" was failing to mount masterserverIP:/var/lib/cdsw/projects

the reason was the the incorrect master IP at /etc/cdsw/config/cdsw.conf

Regards.

NES