Member since
05-21-2017
33
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4014 | 06-29-2017 10:25 PM |
07-11-2017
02:12 AM
Below is the output of "kubectl logs pod_id engine" [root@hostname ~]# kubectl logs ubgujdi5b9b6mmwr engine
2017-07-11 09:06:10.097 9 INFO Engine Waiting one second for Spark config... data = {"id":"ubgujdi5b9b6mmwr"}
2017-07-11 09:06:11.186 15 INFO Engine Waiting one second for Spark config... data = {"id":"ubgujdi5b9b6mmwr"}
/var/lib/cdsw/config/startup.sh: line 31: undefined: command not found
2017/07/11 09:06:12 Loading config file at: /var/lib/cdsw/deps/terminal-conf
2017/07/11 09:06:12 Permitting clients to write input to the PTY.
2017/07/11 09:06:12 Server is starting with command: /bin/bash
2017/07/11 09:06:12 URL: http://0.0.0.0:8000/xmfwh64etire9k0l/
2017/07/11 09:06:13 100.66.0.1:51408 301 GET /xmfwh64etire9k0l
2017-07-11 09:06:13.636 7 INFO Engine ubgujdi5b9b6mmwr Start Authenticating to livelog data = {"secondsSinceStartup":0.85}
Livelog Open
2017-07-11 09:06:13.679 7 INFO Engine ubgujdi5b9b6mmwr Finish Authenticating to livelog: success data = {"secondsSinceStartup":0.898}
2017-07-11 09:06:13.680 7 INFO Engine ubgujdi5b9b6mmwr Start Searching for engine module data = {"secondsSinceStartup":0.9}
2017-07-11 09:06:14.410 7 INFO Engine ubgujdi5b9b6mmwr Finish Searching for engine module: success data = {"engineModule_path":"/usr/local/lib/node_modules/python2-engine"}
2017-07-11 09:06:14.410 7 INFO Engine ubgujdi5b9b6mmwr Start Creating engine data = {"secondsSinceStartup":1.63}
PID of parser IPython process is 59
PID of main IPython process is 62
2017-07-11 09:06:14.661 7 INFO Engine ubgujdi5b9b6mmwr Finish Creating engine data = {"secondsSinceStartup":1.88}
2017-07-11 09:06:22.492 7 INFO Engine ubgujdi5b9b6mmwr Start Registering running status data = {"useHttps":false,"host":"100.77.0.130","path":"/api/v1/projects/Krishna/test1/dashboards/ubgujdi5b9b6mmwr/register-status","senseDomain":"cdsw.adobe.com"}
2017-07-11 09:06:22.505 7 INFO Engine ubgujdi5b9b6mmwr Finish Registering running status: success
2017-07-11 09:06:22.506 7 INFO Engine ubgujdi5b9b6mmwr Pod is ready data = {"secondsSinceStartup":9.726,"engineModuleShare":8.096}
... View more
07-10-2017
09:37 PM
Hi Tristan, "kubectl get events" didn't gave any ouput. Below is the "cdsw status" output [root@hostname ~]# cdsw status
Cloudera Data Science Workbench Status
Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...
Node Status
NAME STATUS AGE STATEFUL
master Ready 17h true
worker1 Ready 17h <none>
worker2 Ready 17h <none>
worker3 Ready 17h <none>
worker3 Ready 17h <none>
System Pod status
NAME READY STATUS RESTARTS AGE IP NODE
dummy-2088944543-c4pbg 1/1 Running 0 17h 10.x.x.x master
etcd-master 1/1 Running 0 17h 10.x.x.x master
kube-apiserver-master 1/1 Running 0 17h 10.x.x.x master
kube-controller-manager-master 1/1 Running 0 17h 10.x.x.x master
kube-discovery-1150918428-se35m 1/1 Running 0 17h 10.x.x.x master
kube-dns-3873593988-olmcy 3/3 Running 0 17h 100.66.0.2 master
kube-proxy-cr019 1/1 Running 0 17h 10.x.x.x master
kube-proxy-o316l 1/1 Running 0 17h 10.x.x.x worker3
kube-proxy-txbph 1/1 Running 0 17h 10.x.x.x worker2
kube-proxy-u0riv 1/1 Running 0 17h 10.x.x.x worker3
kube-proxy-xf6ta 1/1 Running 0 17h 10.x.x.x worker1
kube-scheduler-master 1/1 Running 0 17h 10.x.x.x master
node-problem-detector-v0.1-7zp8i 1/1 Running 0 17h 10.x.x.x worker1
node-problem-detector-v0.1-be2cf 1/1 Running 0 17h 10.x.x.x worker3
node-problem-detector-v0.1-ej7yx 1/1 Running 0 17h 10.x.x.x worker2
node-problem-detector-v0.1-maik6 1/1 Running 0 17h 10.x.x.x master
node-problem-detector-v0.1-xf9o0 1/1 Running 0 17h 10.x.x.x worker3
weave-net-31402 2/2 Running 0 17h 10.x.x.x worker1
weave-net-71t9s 2/2 Running 0 17h 10.x.x.x worker3
weave-net-8p26z 2/2 Running 0 17h 10.x.x.x worker3
weave-net-m4e8x 2/2 Running 0 17h 10.x.x.x worker2
weave-net-wfd35 2/2 Running 0 17h 10.x.x.x master
Cloudera Data Science Workbench Pod Status
NAME READY STATUS RESTARTS AGE IP NODE ROLE
cron-2934152315-ymbxu 1/1 Running 0 17h 100.66.0.8 master cron
db-39862959-s2ic8 1/1 Running 0 17h 100.66.0.4 master db
db-migrate-052787a-170ff 0/1 Completed 0 17h 100.66.0.5 master db-migrate
engine-deps-3uqcr 1/1 Running 0 17h 100.66.0.3 master engine-deps
engine-deps-6npbb 1/1 Running 0 17h 100.66.192.1 worker1 engine-deps
engine-deps-m2385 1/1 Running 0 17h 100.66.64.1 worker2 engine-deps
engine-deps-qgcwy 1/1 Running 0 17h 100.66.128.1 worker3 engine-deps
engine-deps-zblkz 1/1 Running 0 17h 100.66.160.1 worker3 engine-deps
ingress-controller-3138093376-nx1wi 1/1 Running 0 17h 10.x.x.x master ingress-controller
livelog-1900214889-bqhf7 1/1 Running 0 17h 100.66.0.7 master livelog
reconciler-459456250-ma02c 1/1 Running 0 17h 100.66.0.6 master reconciler
spark-port-forwarder-0yxno 1/1 Running 0 17h 10.x.x.x worker2 spark-port-forwarder
spark-port-forwarder-86dv2 1/1 Running 0 17h 10.x.x.x worker3 spark-port-forwarder
spark-port-forwarder-l2u4k 1/1 Running 0 17h 10.x.x.x worker1 spark-port-forwarder
spark-port-forwarder-lpwms 1/1 Running 0 17h 10.x.x.x master spark-port-forwarder
spark-port-forwarder-rsx25 1/1 Running 0 17h 10.x.x.x worker3 spark-port-forwarder
web-3826671331-0n92g 1/1 Running 0 17h 100.66.0.10 master web
web-3826671331-my2vs 1/1 Running 0 17h 100.66.0.9 master web
web-3826671331-zva8n 1/1 Running 0 17h 100.66.0.5 master web
Cloudera Data Science Workbench is ready! "kubectl logs <stuck-pod-id> engine" no pod in stuck mod, its stuck while launching te container in WebUI, Kindly check below screenshot. Thanks Krishna
... View more
07-10-2017
03:43 AM
i have tried below, Test 1: cdsw enable "worker_node_ip" Result: Same issue Test 2: Removed nodes from the cluster, added again Result: Same issue Test 3: Reset Master and workers, performed "cdsw init" and "cdsw enable worker_ip" on master and "cdsw join" Result: Same issue Still getting " ContainerCreating: Creating engine container." Admin --> Site Administration--> Overview
... View more
07-06-2017
04:55 AM
Hi Peter, Its not giving any response. It is also kind of stuck. Let me re-check with network admin. Thanks Krishna
... View more
06-29-2017
10:49 PM
While lauching the teminal access for workbook, everytime i'm entering the URI name of the link in hosts file. Does this is the expected behaviour..?
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-29-2017
10:45 PM
I have installed work bench on 5 node cluster. Everything looks good. But when i checked cdsw status its showing stateful as <none> Node Status
NAME STATUS AGE STATEFUL
hostname1 Ready 1h true
hostname2 Ready 1h <none>
hostname3 Ready 1h <none>
hostname4 Ready 1h <none>
hostname5 Ready 1h <none> And when i launched workbench from cdsw.company.com, its forever showing " ContainerCreating: Creating engine container." and input field is blinking in red color
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-29-2017
10:25 PM
I fixed the issue "Cloudera Data Science Workbench is not ready yet: cannot curl localhost" by starting the httpd service. Now every worker node is showing "Cloudera Data Science Workbench is ready!"
... View more
06-29-2017
09:05 PM
@peter_ableda Any update The same installation is working fine in VM's. This i'm doing on physical machines
... View more
06-29-2017
01:40 AM
Earlier i was trying without using NO_PROXY="localhost,127.0.0.1". That time also same issue
... View more
06-29-2017
12:44 AM
Hi Peter, We are not using Proxy and firewall also stopped. While running kubectl cluster-info dump, getting below error [root@hostname cdsw-logs-hostname-2017-06-29--06-49-53]# kubectl cluster-info dump
The connection to the server localhost:8080 was refused - did you specify the right host or port? I have shared a link for cdsw logs files, kindly check your private messages. I hope this will give more infomation. Thanks Krishna
... View more
06-29-2017
12:20 AM
Hi Peter, Still it not moving forward. Below is cdsw status output [root@hostname ~]# cdsw status
Cloudera Data Science Workbench Status
Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...
Node Status
Cloudera Data Science Workbench is not ready yet: kubectl command failed Kindly check my earlier update, while collecting the logs its showing The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
... View more
06-28-2017
11:56 PM
cdsw logs showing below output [root@hostname ~]# cdsw logs
Generating Cloudera Data Science Workbench diagnostic bundle...
Checking system basics...
Saving kernel parameters...
Checking validation output...
Checking application configuration...
Checking disks...
Checking Hadoop configuration...
Checking network...
Checking system services...
Checking Docker...
Checking Kubernetes...
Checking Kubelet...
Checking application services...
Checking cluster info...
Checking app cluster info...
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
Exporting user ids...
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
The connection to the server 10.x.x.x:6443 was refused - did you specify the right host or port?
Checking system logs...
Producing logs tarball...
Logs saved to: cdsw-logs-hostname-2017-06-29--06-49-53.tar.gz
Redacting logs...
Producing redacted logs tarball...
Redacted logs saved to: cdsw-logs-hostname-2017-06-29--06-49-53.redacted.tar.gz
Cleaning up...
[root@hostname ~]#
... View more
06-28-2017
10:34 AM
I'm getting below error Error response from daemon: Get https://gcr.io/v1/_ping: net/http: TLS handshake timeout
... View more
06-28-2017
10:14 AM
below is the output of mentioned commands [root@hostname ~]# systemctl status docker
● docker.service - docker
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-06-28 16:07:41 GMT; 59min ago
Docs: https://docs.docker.com
Main PID: 136608 (dockerd)
Memory: 25.7M
CGroup: /system.slice/docker.service
├─136608 dockerd --log-driver=journald -s devicemapper --storage-opt dm.basesize=100G --storage-opt dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_rem...
└─136625 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/cont...
Jun 28 17:06:53 hostname docker[136608]: time="2017-06-28T17:06:53.813294239Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:06:55 hostname docker[136608]: time="2017-06-28T17:06:55.512839511Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:07:00 hostname docker[136608]: time="2017-06-28T17:07:00.813523993Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:07:01 hostname docker[136608]: time="2017-06-28T17:07:01.986498328Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: T...ke timeout"
Jun 28 17:07:01 hostname docker[136608]: time="2017-06-28T17:07:01.986557602Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.i...ke timeout"
Jun 28 17:07:05 hostname docker[136608]: time="2017-06-28T17:07:05.813261563Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:07:08 hostname docker[136608]: time="2017-06-28T17:07:08.512474126Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:07:11 hostname docker[136608]: time="2017-06-28T17:07:11.813302105Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0...-amd64:3.0"
Jun 28 17:07:12 hostname docker[136608]: time="2017-06-28T17:07:12.078578325Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.i...ke timeout"
Jun 28 17:07:12 hostname docker[136608]: time="2017-06-28T17:07:12.078630585Z" level=error msg="Handler for POST /images/create returned error: Get https://gcr....ke timeout"
Hint: Some lines were ellipsized, use -l to show in full.
[root@hostname ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.repository.cloudera.com/cdsw/1.0.1/third-party/weaveexec 1.9.0 300f92429697 4 months ago 90.4 MB
[root@hostname~]# journalctl -u docker
-- Logs begin at Wed 2017-06-28 12:13:22 GMT, end at Wed 2017-06-28 17:07:39 GMT. --
Jun 28 12:15:34 hostname systemd[1]: Starting docker...
Jun 28 12:15:34 hostname docker[26682]: Command "daemon" is deprecated, and will be removed in Docker 1.16. Please run `dockerd` directly.
Jun 28 12:15:34 hostname docker[26682]: time="2017-06-28T12:15:34.677250601Z" level=info msg="libcontainerd: new containerd process, pid: 26751"
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.871080507Z" level=warning msg="devmapper: Base device already exists and has filesystem xfs on it. User spec
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.919664135Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.920182783Z" level=warning msg="mountpoint for pids not found"
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.920875579Z" level=info msg="Loading containers: start."
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.952827173Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon op
Jun 28 12:15:35 hostname docker[26682]: time="2017-06-28T12:15:35.967289128Z" level=info msg="Loading containers: done." Jun 28 12:15:36 hostname docker[26682]: time="2017-06-28T12:15:36.010477234Z" level=info msg="Daemon has completed initialization" Jun 28 12:15:36 hostname docker[26682]: time="2017-06-28T12:15:36.010527297Z" level=info msg="Docker daemon" commit=49bf474 graphdriver=devicemapper version=1.13.0 Jun 28 12:15:36 hostname systemd[1]: Started docker. Jun 28 12:15:36 hostname docker[26682]: time="2017-06-28T12:15:36.024593232Z" level=info msg="API listen on /var/run/docker.sock" Jun 28 12:15:37 hostname docker[26682]: time="2017-06-28T12:15:37.400330485Z" level=warning msg="mountpoint for pids not found" Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.450533401Z" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not foun Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.518085080Z" level=error msg="Error setting up exec command in container weaveproxy: No such container: weave Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.518139532Z" level=error msg="Handler for POST /v1.22/containers/weaveproxy/exec returned error: No such cont Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.585161570Z" level=error msg="Handler for GET /v1.22/containers/weave/json returned error: No such container: Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.585823308Z" level=error msg="Handler for GET /v1.22/images/weave/json returned error: No such image: weave" Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.645337877Z" level=error msg="Handler for POST /v1.22/containers/weaveplugin/stop returned error: No such con Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.702047141Z" level=error msg="Handler for DELETE /v1.22/containers/weaveplugin returned error: No such contai Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.755125181Z" level=error msg="Handler for POST /v1.22/containers/weave/stop returned error: No such container Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.820383586Z" level=error msg="Handler for DELETE /v1.22/containers/weave returned error: No such container: w Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.866553352Z" level=error msg="Handler for POST /v1.22/containers/weaveproxy/stop returned error: No such cont Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.934899070Z" level=error msg="Handler for DELETE /v1.22/containers/weaveproxy returned error: No such contain Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.991199157Z" level=error msg="Handler for GET /v1.22/containers/weaveplugin/json returned error: No such cont Jun 28 12:15:38 hostname docker[26682]: time="2017-06-28T12:15:38.991688746Z" level=error msg="Handler for GET /v1.22/images/weaveplugin/json returned error: No such image: w Jun 28 12:15:39 hostname docker[26682]: time="2017-06-28T12:15:39.945499212Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 55fa740127dbb Jun 28 12:15:52 hostname docker[26682]: time="2017-06-28T12:15:52.260203323Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:15:52 hostname docker[26682]: time="2017-06-28T12:15:52.554632409Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:15:52 hostname docker[26682]: time="2017-06-28T12:15:52.555863242Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:15:52 hostname docker[26682]: time="2017-06-28T12:15:52.557338775Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:16:02 hostname docker[26682]: time="2017-06-28T12:16:02.458600197Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: TLS handshake ti Jun 28 12:16:02 hostname docker[26682]: time="2017-06-28T12:16:02.458659007Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: net/http Jun 28 12:16:12 hostname docker[26682]: time="2017-06-28T12:16:12.551326574Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: net Jun 28 12:16:12 hostname docker[26682]: time="2017-06-28T12:16:12.551407914Z" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: ne Jun 28 12:16:22 hostname docker[26682]: time="2017-06-28T12:16:22.724526575Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: TLS handshake ti Jun 28 12:16:22 hostname docker[26682]: time="2017-06-28T12:16:22.724587795Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: net/http Jun 28 12:16:26 hostname docker[26682]: time="2017-06-28T12:16:26.250424062Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:16:32 hostname docker[26682]: time="2017-06-28T12:16:32.817042449Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: net Jun 28 12:16:32 hostname docker[26682]: time="2017-06-28T12:16:32.817107756Z" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: ne Jun 28 12:16:42 hostname docker[26682]: time="2017-06-28T12:16:42.990655817Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: TLS handshake ti Jun 28 12:16:42 hostname docker[26682]: time="2017-06-28T12:16:42.990714640Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: net/http Jun 28 12:16:45 hostname docker[26682]: time="2017-06-28T12:16:45.550698377Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:16:53 hostname docker[26682]: time="2017-06-28T12:16:53.083222674Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: net Jun 28 12:16:53 hostname docker[26682]: time="2017-06-28T12:16:53.083296701Z" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: ne Jun 28 12:17:03 hostname docker[26682]: time="2017-06-28T12:17:03.257122769Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: TLS handshake ti Jun 28 12:17:03 hostname docker[26682]: time="2017-06-28T12:17:03.257220350Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: net/http Jun 28 12:17:07 hostname docker[26682]: time="2017-06-28T12:17:07.550742117Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:17:13 hostname docker[26682]: time="2017-06-28T12:17:13.349490853Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: net Jun 28 12:17:13 hostname docker[26682]: time="2017-06-28T12:17:13.349596870Z" level=error msg="Handler for POST /images/create returned error: Get https://gcr.io/v1/_ping: ne Jun 28 12:17:23 hostname docker[26682]: time="2017-06-28T12:17:23.522568083Z" level=warning msg="Error getting v2 registry: Get https://gcr.io/v2/: net/http: TLS handshake ti Jun 28 12:17:23 hostname docker[26682]: time="2017-06-28T12:17:23.522623816Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v2/: net/http Jun 28 12:17:25 hostname docker[26682]: time="2017-06-28T12:17:25.550683773Z" level=error msg="Handler for GET /images/gcr.io/google_containers/pause-amd64:3.0/json returned Jun 28 12:17:33 hostname docker[26682]: time="2017-06-28T12:17:33.615292807Z" level=error msg="Attempting next endpoint for pull after error: Get https://gcr.io/v1/_ping: net lines 1-56
... View more
06-28-2017
09:20 AM
Hi Peter, I started building new cluster, while doing that, i got below issue. http://community.cloudera.com/t5/Cloudera-Data-Science-Workbench/cdsw-init-failed/m-p/56646#M84
... View more
06-28-2017
09:19 AM
This is on my fresh servers, after installing cloudera-workbench rpm, while doing "cdsw init", i got below [root@hostname ~]# cdsw init
Using user-specified config file: /etc/cdsw/config/cdsw.conf
Prechecking OS Version........[OK]
Prechecking scaling limits for processes........[OK]
Prechecking scaling limits for open files........
WARNING: Cloudera Data Science Workbench recommends that all users have a max-open-files limit set to 1048576.
It is currently set to [65535] as per 'ulimit -n'
Press enter to continue
Prechecking that iptables are not configured........
WARNING: Cloudera Data Science Workbench requires iptables, but does not support preexisting iptables rules. Press enter to continue
Prechecking that SELinux is disabled........[OK]
Prechecking configured block devices and mountpoints........[OK]
Prechecking kernel parameters........[OK]
Prechecking that docker block devices are of adequate size........[OK]
Prechecking that application block devices are of adequate size........[OK]
Prechecking size of root volume........
WARNING: The recommended minimum root volume size is 100G. Press enter to continue
Prechecking that CDH gateway roles are configured........[OK]
Prechecking that /etc/krb5 file is not a placeholder........
WARNING: The Kerberos configuration file [/etc/krb5.conf] seems to be a placeholder. If your CDH cluster is Kerberized, please copy /etc/krb5.conf to these Cloudera Data Science Workbench nodes.
Press enter to continue.
Prechecking parcel paths........
WARNING: CDH parcels not found at /opt/cloudera/parcels. If you are using a custom parcel directory,
please set it in the Cloudera Data Science Workbench admin panel once the site is running. Otherwise,
please add your Cloudera Data Science Workbench nodes to your CDH cluster.
Press enter to continue.
Prechecking CDH client configurations........
WARNING: CDH client configuration not found at /etc/spark2-conf. Press enter to continue
Prechecking Java version........[OK]
Prechecking Java distribution........
WARNING: OpenJDK is not supported. Press enter to continue
Creating docker thinpool if it does not exist
--- Logical volume ---
LV Name thinpool
VG Name docker
LV UUID E0BUAz-vz2P-DoPB-5u9w-Zjjl-wPhQ-deiZqt
LV Write Access read/write
LV Creation host, time hostname, 2017-06-28 11:08:49 +0000
LV Pool metadata thinpool_tmeta
LV Pool data thinpool_tdata
LV Status available
# open 0
LV Size 5.19 TiB
Allocated pool data 0.00%
Allocated metadata 0.02%
Current LE 1359251
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:6
Docker thinpool already configured.
Initialize application storage at /var/lib/cdsw
Disabling node with IP [10.*.*.*]...
Node [10.*.*.*] removed from nfs export list successfully.
Stopping rpc-statd...
Stopping nfs-idmapd...
Stopping rpcbind...
Stopping nfs-server...
Removing entry from /etc/fstab...
Unmounting [/dev/sdc1]...
Skipping format since volumes are already set correctly.
Adding entry to /etc/fstab...
Mounting [/var/lib/cdsw]...
Starting rpc-statd...
Enabling rpc-statd...
Starting nfs-idmapd...
Enabling nfs-idmapd...
Starting rpcbind...
Enabling rpcbind...
Starting nfs-server...
Enabling nfs-server...
Enabling node with IP [10.*.*.*]...
Node [10.*.*.*] added to nfs export list successfully.
Starting rpc-statd...
Enabling rpc-statd...
Starting nfs-idmapd...
Enabling nfs-idmapd...
Starting rpcbind...
Enabling rpcbind...
Starting nfs-server...
Enabling nfs-server...
Starting docker...
Enabling docker...
Starting ntpd...
Enabling ntpd...
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
Initializing cluster...
Running pre-flight checks
<master/tokens> generated token: "1df27c.244acfe17e00a402"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2017-06-28 16:07:45 +0000 UTC Not After: 2027-06-26 16:07:45 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2017-06-28 16:07:45 +0000 UTC Not After: 2018-06-28 16:07:46 +0000 UTC
Alternate Names: [10.*.*.* 100.77.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready Its stopped at created API client, waiting for the control plane to become ready Can anyone help with this
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-21-2017
05:58 AM
I am trying to run "analysis.py", its throwing below error cdsw@9nmx4v1iq3r6thaz:~$ python analysis.py
Traceback (most recent call last):
File "analysis.py", line 5, in <module>
import matplotlib.pyplot as plt
File "/usr/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 115, in <module>
_backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
globals(),locals(),[backend_name],0)
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 18, in <module>
from .backend_qt5agg import FigureCanvasQTAggBase as _FigureCanvasQTAggBase
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/backend_qt5agg.py", line 16, in <module>
from .backend_qt5 import QtCore
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/backend_qt5.py", line 26, in <module>
import matplotlib.backends.qt_editor.figureoptions as figureoptions
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/qt_editor/figureoptions.py", line 20, in <module>
import matplotlib.backends.qt_editor.formlayout as formlayout
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/qt_editor/formlayout.py", line 56, in <module>
from matplotlib.backends.qt_compat import QtGui, QtWidgets, QtCore
File "/usr/local/lib/python2.7/site-packages/matplotlib/backends/qt_compat.py", line 175, in <module>
"Matplotlib qt-based backends require an external PyQt4, PyQt5,\n"
ImportError: Matplotlib qt-based backends require an external PyQt4, PyQt5,
or PySide package to be installed, but it was not found. So i try to install PySide using pip it sthrowing below error. cdsw@9nmx4v1iq3r6thaz:~$ pip install pyside
Collecting pyside
Using cached PySide-1.2.4.tar.gz
Building wheels for collected packages: pyside
Running setup.py bdist_wheel for pyside ... error
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-IGd3Xz/pyside/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpo3P57Spip-wheel- --python-tag cp27:
Removing /tmp/pip-build-IGd3Xz/pyside/pyside_package
running bdist_wheel
running build
Python architecture is 64bit
error: Failed to find qmake. Please specify the path to qmake with --qmake parameter.
----------------------------------------
Failed building wheel for pyside
Running setup.py clean for pyside
Failed to build pyside
Installing collected packages: pyside
Running setup.py install for pyside ... error
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-IGd3Xz/pyside/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n'
, '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-qb6KhV-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
Removing /tmp/pip-build-IGd3Xz/pyside/pyside_package
running install
running build
Python architecture is 64bit
error: Failed to find qmake. Please specify the path to qmake with --qmake parameter.
----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-IGd3Xz/pyside/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(c
ompile(code, __file__, 'exec'))" install --record /tmp/pip-qb6KhV-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-IGd3Xz/pysid
e/ So again i tried installing using "sudo apt-get install python-pyside", again permission error cdsw@9nmx4v1iq3r6thaz:~$ apt-get install python-pyside
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root? So i tried switching as root, it dodn't work, its throwing cdsw@9nmx4v1iq3r6thaz:~$ sudo su
The program 'sudo' can be found in the following packages:
* sudo
* sudo-ldap
Ask your administrator to install one of them How to fix this P.S: I am admin
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-21-2017
04:56 AM
On Master node "cdsw status" is sowing "Cloudera Data Science Workbench is ready!". Below is the output of "cdsw status" command Cloudera Data Science Workbench Status
Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...
Node Status
NAME STATUS AGE STATEFUL
abcd.abc.com Ready 30d true
xxxx.corp.abc Ready 5d <none>
xxxx.corp.abc Ready 1d <none>
xxxx.corp.abc Ready 1d <none>
xxxx.corp.abc Ready 1d <none>
System Pod status
NAME READY STATUS RESTARTS AGE
dummy-2088944543-iu5em 1/1 Running 5 30d
etcd-abcd.corp.abc 1/1 Running 5 30d
kube-apiserver-abcd.corp.abc 1/1 Running 6 30d
kube-controller-manager-abcd.corp.abc 1/1 Running 5 30d
kube-discovery-1150918428-s7m6e 0/1 MatchNodeSelector 0 30d
kube-discovery-1150918428-tfch1 1/1 Running 3 7d
kube-dns-3873593988-g83ve 3/3 Running 15 30d
kube-proxy-2rfvu 1/1 Running 0 1d
kube-proxy-mq6z1 1/1 Running 0 5d
kube-proxy-orp04 1/1 Running 0 1d
kube-proxy-pd3kl 1/1 Running 0 1d
kube-proxy-wlaqj 1/1 Running 5 30d
kube-scheduler-abcd.corp.abc 1/1 Running 5 30d
node-problem-detector-v0.1-bte1v 1/1 Running 0 1d
node-problem-detector-v0.1-cvwav 1/1 Running 0 5d
node-problem-detector-v0.1-extu9 1/1 Running 0 1d
node-problem-detector-v0.1-qlz7s 1/1 Running 5 30d
node-problem-detector-v0.1-vftvo 1/1 Running 0 1d
weave-net-38alm 2/2 Running 11 30d
weave-net-4mg1p 2/2 Running 0 1d
weave-net-e99uh 2/2 Running 0 1d
weave-net-eyern 2/2 Running 1 5d
weave-net-i428d 2/2 Running 0 1d
Cloudera Data Science Workbench Pod Status
NAME READY STATUS RESTARTS AGE ROLE
cron-3971587342-670nl 1/1 Running 5 30d cron
db-4066525870-0xmz5 1/1 Running 3 7d db
db-4066525870-g493s 0/1 MatchNodeSelector 0 30d db
db-migrate-abec968-2pbnb 0/1 Completed 0 30d db-migrate
engine-deps-gz18i 1/1 Running 5 30d engine-deps
engine-deps-jeu62 1/1 Running 0 1d engine-deps
engine-deps-pop4r 1/1 Running 0 1d engine-deps
engine-deps-t7a9m 1/1 Running 0 5d engine-deps
engine-deps-v16a5 1/1 Running 0 1d engine-deps
ingress-controller-2976678207-lrdp8 0/1 MatchNodeSelector 0 30d ingress-controller
ingress-controller-2976678207-qrz8x 1/1 Running 3 7d ingress-controller
livelog-2494298876-22gbi 1/1 Running 3 7d livelog
livelog-2494298876-rhtg5 0/1 MatchNodeSelector 0 30d livelog
reconciler-577027981-r4vni 1/1 Running 5 30d reconciler
spark-port-forwarder-cmcso 1/1 Running 0 1d spark-port-forwarder
spark-port-forwarder-e6a30 1/1 Running 0 1d spark-port-forwarder
spark-port-forwarder-o50lr 1/1 Running 0 1d spark-port-forwarder
spark-port-forwarder-salu1 1/1 Running 0 5d spark-port-forwarder
spark-port-forwarder-tjhaa 1/1 Running 5 30d spark-port-forwarder
web-1304125449-5qb5e 1/1 Running 5 30d web
web-1304125449-na1av 1/1 Running 5 30d web
web-1304125449-qhs08 1/1 Running 5 30d web
Cloudera Data Science Workbench is ready! Note: abcd.abc.com is the master
... View more
06-21-2017
04:36 AM
I have sucessfully added my worker node to master using "cdsw join". Some of the nodes are working fine, some of the node not responding properly. When i typed "cdsw status" i'm getting "Cloudera Data Science Workbench is not ready yet: cannot curl localhost" Thanks Krishna
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-16-2017
03:30 AM
Tried same earlier, that time it didn't work. Now executed again. Its working
... View more
06-16-2017
02:38 AM
Hi, I installed workbench on Master sucessfully, but when i try to add worker nodes to Master i'm getting below error. Could anyone help to fix it. Thanks Krishna
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
06-16-2017
02:35 AM
Hi All, I searching for complete and clean un-installation of workbench to re-install. But i'm not able to find any document. I tried just uninstalling by removing the "cloudera-data-science-workbench-1.0.0-1.el7.centos.x86_64.rpm", but still i'm able to see old configuration files and dependencies. Thanks Krishna
... View more
Labels:
- Labels:
-
Cloudera Data Science Workbench
05-29-2017
03:25 AM
Could you please explain how did you ran test, if possible could you update those details here
... View more
05-23-2017
04:20 AM
1 Kudo
You can also find by doing "locate piggybank.jar"
... View more
05-22-2017
02:01 AM
It should be in "/opt/cloudera/parcels/CDH/lib/pig/"
... View more
05-22-2017
01:59 AM
IP file..? If it is regarding iptables. You can check this by doing "iptables -L" as root. Note: Firewall and iptables should be disabled while installing CDH
... View more