Created on 10-08-2019 04:05 AM - last edited on 10-08-2019 08:00 AM by ask_bill_brooks
HI Guys,
I upgrade to the latest CDSW 1.6 over the weekend. The upgrade was successful, however i am having trouble lunching a CLI terminal once i have created a project. Every time i do, i get the below error.
tty-8c51gbduvszddduehl.domain.co.za is currently unable to handle this request.
Created 10-09-2019 05:46 AM
Hi
Please check if you have faced any errors like below in the CDSW logs like below after upgrade to 1.6
ERROR EngineInit.BrowserSvcs fgtugi359w4b5n58 Finish apiGet, failed to execute API request data = {"err":"Get https://cdsw.unedic.intra/api/v1/projects/Priseenmain/formation_python/engines/fgtugi359w4b5n58/perm...: x509: certificate signed by unknown authority","user":"root"}
The method by which the Cloudera Data Science Workbench web UI session tokens are stored has been hardened.
as the issue happened after the upgrade to CDSW 1.6 I suspect that it is related to the issue addressed under "DSE-7173". It seems that in CDSW 1.6 all internal traffic is et to go through the ingress-controller proxy for more security, however, it seems that some certificates are not making it into the sessions
A possible solution can be by manually copy the root CA and intermediate CA into the underlying cert file that contains all of the OS level certs.
Check if below steps works
Please find the steps below, try it and let us know how it goes.
1. Copy all of the certificates you are using in your chain of trust (root CA, intermediate CA's). It should start and end with ----BEGIN CERTIFICATE----, ----END CERTIFICATE----)
2. Create a session like normal and copy the engineID from the URl. Then figure out which host the pod is running on (last column) with:
# kubectl get po --all-namespaces=true -o wide
3. SSH into that host and run:
docker ps | grep -i <engineID> (to get the container id)
docker cp <containerID>:/etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
This step will copy the ca-certificate file out of the docker container. This file contains all of the certificates that are trusted by the OS.
4. Append ALL of your internal root CA / intermediate CA certificates to the end of this file. Copy this file to all of the CDSW nodes that you have
5. Go back to CDSW -> Admin -> Engines, and under Mounts, add /etc/ssl/certs/ca-certificates.crt to the engine mount. This will ensure that the changes persist across restarts.
6. Create a brand new session in CDSW and ensure that you can open the _Terminal and access Jupyter workbench.
Created 10-09-2019 05:46 AM
Hi
Please check if you have faced any errors like below in the CDSW logs like below after upgrade to 1.6
ERROR EngineInit.BrowserSvcs fgtugi359w4b5n58 Finish apiGet, failed to execute API request data = {"err":"Get https://cdsw.unedic.intra/api/v1/projects/Priseenmain/formation_python/engines/fgtugi359w4b5n58/perm...: x509: certificate signed by unknown authority","user":"root"}
The method by which the Cloudera Data Science Workbench web UI session tokens are stored has been hardened.
as the issue happened after the upgrade to CDSW 1.6 I suspect that it is related to the issue addressed under "DSE-7173". It seems that in CDSW 1.6 all internal traffic is et to go through the ingress-controller proxy for more security, however, it seems that some certificates are not making it into the sessions
A possible solution can be by manually copy the root CA and intermediate CA into the underlying cert file that contains all of the OS level certs.
Check if below steps works
Please find the steps below, try it and let us know how it goes.
1. Copy all of the certificates you are using in your chain of trust (root CA, intermediate CA's). It should start and end with ----BEGIN CERTIFICATE----, ----END CERTIFICATE----)
2. Create a session like normal and copy the engineID from the URl. Then figure out which host the pod is running on (last column) with:
# kubectl get po --all-namespaces=true -o wide
3. SSH into that host and run:
docker ps | grep -i <engineID> (to get the container id)
docker cp <containerID>:/etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
This step will copy the ca-certificate file out of the docker container. This file contains all of the certificates that are trusted by the OS.
4. Append ALL of your internal root CA / intermediate CA certificates to the end of this file. Copy this file to all of the CDSW nodes that you have
5. Go back to CDSW -> Admin -> Engines, and under Mounts, add /etc/ssl/certs/ca-certificates.crt to the engine mount. This will ensure that the changes persist across restarts.
6. Create a brand new session in CDSW and ensure that you can open the _Terminal and access Jupyter workbench.
Created 10-09-2019 05:48 AM
I actually found that under the known limitation of CDSW 1.6 and applied the workaround and worked.
Thanks