Support Questions

Find answers, ask questions, and share your expertise

Cloudera Data Science Workbench 1.6 latest.

avatar
Explorer

HI Guys,

 

I upgrade to the latest CDSW 1.6 over the weekend. The upgrade was successful, however i am having trouble lunching a CLI terminal once i have created a project. Every time i do, i get the below error.   

 

This page isn’t working

tty-8c51gbduvszddduehl.domain.co.za is currently unable to handle this request.

HTTP ERROR 500
 
My cluster is fully TLS'ed and only having this problem since the upgrade. 
1 ACCEPTED SOLUTION

avatar
Rising Star

Hi 

Please check if you have faced any errors like below in the CDSW logs like below after upgrade to 1.6

ERROR EngineInit.BrowserSvcs fgtugi359w4b5n58 Finish apiGet, failed to execute API request data = {"err":"Get https://cdsw.unedic.intra/api/v1/projects/Priseenmain/formation_python/engines/fgtugi359w4b5n58/perm...: x509: certificate signed by unknown authority","user":"root"}

The method by which the Cloudera Data Science Workbench web UI session tokens are stored has been hardened.

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_release_notes.html#...

 

as the issue happened after the upgrade to CDSW 1.6 I suspect that it is related to the issue addressed under "DSE-7173". It seems that in CDSW 1.6 all internal traffic is et to go through the ingress-controller proxy for more security, however, it seems that some certificates are not making it into the sessions

A possible solution can be by manually copy the root CA and intermediate CA into the underlying cert file that contains all of the OS level certs.

Check if below steps works

Please find the steps below, try it and let us know how it goes.

1. Copy all of the certificates you are using in your chain of trust (root CA, intermediate CA's). It should start and end with ----BEGIN CERTIFICATE----, ----END CERTIFICATE----)
2. Create a session like normal and copy the engineID from the URl. Then figure out which host the pod is running on (last column) with:
# kubectl get po --all-namespaces=true -o wide
3. SSH into that host and run:
docker ps | grep -i <engineID> (to get the container id)
docker cp <containerID>:/etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
This step will copy the ca-certificate file out of the docker container. This file contains all of the certificates that are trusted by the OS.
4. Append ALL of your internal root CA / intermediate CA certificates to the end of this file. Copy this file to all of the CDSW nodes that you have
5. Go back to CDSW -> Admin -> Engines, and under Mounts, add /etc/ssl/certs/ca-certificates.crt to the engine mount. This will ensure that the changes persist across restarts.
6. Create a brand new session in CDSW and ensure that you can open the _Terminal and access Jupyter workbench.









View solution in original post

2 REPLIES 2

avatar
Rising Star

Hi 

Please check if you have faced any errors like below in the CDSW logs like below after upgrade to 1.6

ERROR EngineInit.BrowserSvcs fgtugi359w4b5n58 Finish apiGet, failed to execute API request data = {"err":"Get https://cdsw.unedic.intra/api/v1/projects/Priseenmain/formation_python/engines/fgtugi359w4b5n58/perm...: x509: certificate signed by unknown authority","user":"root"}

The method by which the Cloudera Data Science Workbench web UI session tokens are stored has been hardened.

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_release_notes.html#...

 

as the issue happened after the upgrade to CDSW 1.6 I suspect that it is related to the issue addressed under "DSE-7173". It seems that in CDSW 1.6 all internal traffic is et to go through the ingress-controller proxy for more security, however, it seems that some certificates are not making it into the sessions

A possible solution can be by manually copy the root CA and intermediate CA into the underlying cert file that contains all of the OS level certs.

Check if below steps works

Please find the steps below, try it and let us know how it goes.

1. Copy all of the certificates you are using in your chain of trust (root CA, intermediate CA's). It should start and end with ----BEGIN CERTIFICATE----, ----END CERTIFICATE----)
2. Create a session like normal and copy the engineID from the URl. Then figure out which host the pod is running on (last column) with:
# kubectl get po --all-namespaces=true -o wide
3. SSH into that host and run:
docker ps | grep -i <engineID> (to get the container id)
docker cp <containerID>:/etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
This step will copy the ca-certificate file out of the docker container. This file contains all of the certificates that are trusted by the OS.
4. Append ALL of your internal root CA / intermediate CA certificates to the end of this file. Copy this file to all of the CDSW nodes that you have
5. Go back to CDSW -> Admin -> Engines, and under Mounts, add /etc/ssl/certs/ca-certificates.crt to the engine mount. This will ensure that the changes persist across restarts.
6. Create a brand new session in CDSW and ensure that you can open the _Terminal and access Jupyter workbench.









avatar
Explorer

I actually found that under the known limitation of CDSW 1.6 and applied the workaround and worked.

 

Thanks