Re: CDSW

Mike · ‎03-30-2022

When this is happening, are you able to start Sessions as well? Do you have access to the Admin -> Usage page or kubectl access? You should look to see if there are enough resources available for the engine that you have chosen to use when running the job.

Mike · ‎11-02-2021

Sure you are welcome. It is definitely an interesting topic but it's pretty hard to get some actual data, so much depends on the type of workloads you want to run, the size of your nodes, etc. Good luck!

Mike · ‎11-02-2021

Hi, there is not a hard limit on the number of CDSW worker nodes you can have, however there are practical limits based - if you have say thirty nodes, there starts to be a lot more overhead in terms of network traffic and latency. For instance, each worker node will require about 3cpu and 5gb of ram just for the kubelet and internal CDSW pods - so if you have 30 worker nodes, you will be loosing 90cpu and 150gb of ram, which might not pay off. On larger clusters there is a delicate balance between how big your worker nodes are and how many worker nodes you choose to have - I can't really give much guidance on here other than it takes some trial and error to get right. If you have an account with Cloudera you should reach out to that team to get some more detailed information. Some rough guidelines would be to have workers between 32 and 64 vCPU, and have less than 20 of them....but, your mileage may vary. Hope this helps.

Mike · ‎04-01-2021

It looks like you have CDSW configured to use /dev/sdb for your docker block device: ERROR:: Error in pvcreate for [/dev/sdb]: 5 But somehow you also have /dev/sde set: ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/sde]: 1 Check your configuration for DOCKER_BLOCK_DEVICE in CM, it is possible that you have multiple disks set for some reason. (People often do this when they have multiple nodes and the disks are different on each node. You need to use the role group feature to do this instead of listing all of the disks.) A docker block device will be created on the master AND worker, so you should make sure that both machines hav a /dev/sde (or /dev/sdb) that is free and formatted. OR use role groups to set master to be dev/sdb and worker to be /dev/sde. > Also, please let me know if setting up DNS wildcard is adding the *domain name on /etc/hosts or any other DNS file of the linux server No. You need to go into the DNS server itself and add an entry for a wildcard DNS for your domain. It should be *.[your-cdsw-url]. You cannot use /etc/host entries; CDSW ignores /etc/host in containers. I have never set up a DNS server so I can't really help with this.

Mike · ‎04-01-2021

Did you run the Prepare Node step?

Mike · ‎03-16-2021

This is a pretty interesting question, at first I was going to suggest just using a copy command in your Dockerfile to copy this file over, however, I'm not totally positive that the .jupyter directory exists until you start up a CDSW session with .jupyter notebook. Can you show me what you are adding in the config file to time out the Jupyter notebooks? You are correct that Jupyter notebooks do not time out from the IDLE_MAXIMUM_MINUTES environment variable. R Studio sessions do not either and this has been a long running and difficult issue since Cloudera doesn't write or control this code. It looks like a lot of this is fixed in CDSW 1.9 though. If you just want to time out. jupyter notebooks, you could try to edit the Jupyter Notebook command and add this: NOTEBOOK_TIMEOUT_SECONDS=$(python3 -c "print(${IDLE_MAXIMUM_MINUTES}*60)" ) /usr/local/bin/jupyter notebook --no-browser --ip=127.0.0.1 --port=${CDSW_APP_PORT} --NotebookApp.token= --NotebookApp.allow_remote_access=True --NotebookApp.quit_button=False --log-level=ERROR --NotebookApp.shutdown_no_activity_timeout=300 --MappingKernelManager.cull_idle_timeout=${NOTEBOOK_TIMEOUT_SECONDS} -- TerminalManager.cull_inactive_timeout=${NOTEBOOK_TIMEOUT_SECONDS} --MappingKernelManager.cull_interval=60 --TerminalManager.cull_interval=60 --MappingKernelManager.cull_connected=True This will kill Jupyter Notebooks that have been longer than IDLE_MAXIMUM_MINUTES of inactivity (default to 60 minutes.) There are a few caveats to this, the main one being that this still wont kill Jupyter Terminals due to the version of Jupyter Notebooks that CDSW 1.7 uses. Also, users will not get a warning; their Notebook and corresponding CDSW session will just get killed. You can try this and let me know if it works. I'm also curious about your config file that you want to add into .jupyter.

Mike · ‎03-16-2021

Hey sorry for the delay in getting back to you. You don't need to have spark installed on the CDSW nodes, but you DO need to have the Spark Gateway roles installed. This is how CDSW picks up the configuration for Spark and knows which jar files to use, etc. If you have this working with Python, does that mean you can run the "pi.py" example from a Python session which uses Spark? If so then the problem is not the gateways. If spark is not installed then yes I think this is probably the problem since the Scala workbench tries to create a SparkSession object immediately. This might be why your stack trace has lots of spark paths in it near the end.

Mike · ‎02-10-2021

Hello, are you still having issues with this? What engine image are you using? Can you ensure that there are Spark gateways on your CDSW nodes? The Scala kernel is different than the Python and R kernels in that it makes a connection to Spark upon start up instead of waiting until you create the spark context.

Mike · ‎02-10-2021

Hello, I am glad you were able to figure this out. Just to clarify, IE is not supported in CDSW as of CDSW 1.7+. You can view the list of supported browsers here: https://docs.cloudera.com/documentation/data-science-workbench/1-7-x/topics/cdsw_requirements_supported_versions.html#browser_support

Mike · ‎11-12-2019

Hi, the link you have provided is the recommended way to pull large data sets into your CDSW environment. You can also use git to pull in data from remote servers if they are stored in a git repository: https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_using_git.html Can you talk more about what you are trying to do? If you are just trying to pull in a file on a server somewhere, you should make sure you can ping that file first from the workbench, than you can just download it with whatever your favorite python/scala library is. For example, this tutorial shows how to make a simple website skimmer in Python to pull data off public web pages: https://www.digitalocean.com/community/tutorials/how-to-scrape-web-pages-with-beautiful-soup-and-python-3

Online	Offline
Last Visited	‎07-02-2022 09:22 AM

Date Registered	‎12-21-2017 12:02 PM
Last Visited	‎07-02-2022 09:22 AM
Total Messages Posted	47

Re: CDSW

Re: Job status Scheduling in Cloudera

Re: CDSW

Re: CDSW

Re: issue creating disk Docker image Block device,...

Re: Need help in clearing errors of new CDSW setup

Re: Add file to base CDSW image build.

Re: CDSW Scala workbench error from toree

Re: CDSW Scala workbench error from toree

Re: Tour of CDSW Tutorial Crash

Re: How to import a file from Remote server to CDS...