Member since
12-21-2017
47
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
253 | 11-02-2021 08:28 AM |
03-30-2022
03:33 PM
When this is happening, are you able to start Sessions as well? Do you have access to the Admin -> Usage page or kubectl access? You should look to see if there are enough resources available for the engine that you have chosen to use when running the job.
... View more
11-02-2021
08:53 AM
Sure you are welcome. It is definitely an interesting topic but it's pretty hard to get some actual data, so much depends on the type of workloads you want to run, the size of your nodes, etc. Good luck!
... View more
11-02-2021
08:28 AM
Hi, there is not a hard limit on the number of CDSW worker nodes you can have, however there are practical limits based - if you have say thirty nodes, there starts to be a lot more overhead in terms of network traffic and latency. For instance, each worker node will require about 3cpu and 5gb of ram just for the kubelet and internal CDSW pods - so if you have 30 worker nodes, you will be loosing 90cpu and 150gb of ram, which might not pay off. On larger clusters there is a delicate balance between how big your worker nodes are and how many worker nodes you choose to have - I can't really give much guidance on here other than it takes some trial and error to get right. If you have an account with Cloudera you should reach out to that team to get some more detailed information. Some rough guidelines would be to have workers between 32 and 64 vCPU, and have less than 20 of them....but, your mileage may vary. Hope this helps.
... View more
04-01-2021
06:41 AM
It looks like you have CDSW configured to use /dev/sdb for your docker block device: ERROR:: Error in pvcreate for [/dev/sdb]: 5 But somehow you also have /dev/sde set: ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/sde]: 1 Check your configuration for DOCKER_BLOCK_DEVICE in CM, it is possible that you have multiple disks set for some reason. (People often do this when they have multiple nodes and the disks are different on each node. You need to use the role group feature to do this instead of listing all of the disks.) A docker block device will be created on the master AND worker, so you should make sure that both machines hav a /dev/sde (or /dev/sdb) that is free and formatted. OR use role groups to set master to be dev/sdb and worker to be /dev/sde. > Also, please let me know if setting up DNS wildcard is adding the *domain name on /etc/hosts or any other DNS file of the linux server No. You need to go into the DNS server itself and add an entry for a wildcard DNS for your domain. It should be *.[your-cdsw-url]. You cannot use /etc/host entries; CDSW ignores /etc/host in containers. I have never set up a DNS server so I can't really help with this.
... View more
04-01-2021
06:32 AM
Did you run the Prepare Node step?
... View more
03-16-2021
06:27 AM
This is a pretty interesting question, at first I was going to suggest just using a copy command in your Dockerfile to copy this file over, however, I'm not totally positive that the .jupyter directory exists until you start up a CDSW session with .jupyter notebook. Can you show me what you are adding in the config file to time out the Jupyter notebooks? You are correct that Jupyter notebooks do not time out from the IDLE_MAXIMUM_MINUTES environment variable. R Studio sessions do not either and this has been a long running and difficult issue since Cloudera doesn't write or control this code. It looks like a lot of this is fixed in CDSW 1.9 though. If you just want to time out. jupyter notebooks, you could try to edit the Jupyter Notebook command and add this: NOTEBOOK_TIMEOUT_SECONDS=$(python3 -c "print(${IDLE_MAXIMUM_MINUTES}*60)" ) /usr/local/bin/jupyter notebook --no-browser --ip=127.0.0.1 --port=${CDSW_APP_PORT} --NotebookApp.token= --NotebookApp.allow_remote_access=True --NotebookApp.quit_button=False --log-level=ERROR --NotebookApp.shutdown_no_activity_timeout=300 --MappingKernelManager.cull_idle_timeout=${NOTEBOOK_TIMEOUT_SECONDS} -- TerminalManager.cull_inactive_timeout=${NOTEBOOK_TIMEOUT_SECONDS} --MappingKernelManager.cull_interval=60 --TerminalManager.cull_interval=60 --MappingKernelManager.cull_connected=True This will kill Jupyter Notebooks that have been longer than IDLE_MAXIMUM_MINUTES of inactivity (default to 60 minutes.) There are a few caveats to this, the main one being that this still wont kill Jupyter Terminals due to the version of Jupyter Notebooks that CDSW 1.7 uses. Also, users will not get a warning; their Notebook and corresponding CDSW session will just get killed. You can try this and let me know if it works. I'm also curious about your config file that you want to add into .jupyter.
... View more
03-16-2021
06:17 AM
Hey sorry for the delay in getting back to you. You don't need to have spark installed on the CDSW nodes, but you DO need to have the Spark Gateway roles installed. This is how CDSW picks up the configuration for Spark and knows which jar files to use, etc. If you have this working with Python, does that mean you can run the "pi.py" example from a Python session which uses Spark? If so then the problem is not the gateways. If spark is not installed then yes I think this is probably the problem since the Scala workbench tries to create a SparkSession object immediately. This might be why your stack trace has lots of spark paths in it near the end.
... View more
02-10-2021
05:40 AM
Hello, are you still having issues with this? What engine image are you using? Can you ensure that there are Spark gateways on your CDSW nodes? The Scala kernel is different than the Python and R kernels in that it makes a connection to Spark upon start up instead of waiting until you create the spark context.
... View more
02-10-2021
05:27 AM
Hello, I am glad you were able to figure this out. Just to clarify, IE is not supported in CDSW as of CDSW 1.7+. You can view the list of supported browsers here: https://docs.cloudera.com/documentation/data-science-workbench/1-7-x/topics/cdsw_requirements_supported_versions.html#browser_support
... View more
11-12-2019
01:36 PM
Hi, the link you have provided is the recommended way to pull large data sets into your CDSW environment. You can also use git to pull in data from remote servers if they are stored in a git repository: https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_using_git.html Can you talk more about what you are trying to do? If you are just trying to pull in a file on a server somewhere, you should make sure you can ping that file first from the workbench, than you can just download it with whatever your favorite python/scala library is. For example, this tutorial shows how to make a simple website skimmer in Python to pull data off public web pages: https://www.digitalocean.com/community/tutorials/how-to-scrape-web-pages-with-beautiful-soup-and-python-3
... View more