Member since
04-16-2018
3
Posts
0
Kudos Received
0
Solutions
04-16-2018
06:46 AM
@saranvisa Thanks for your quick reply, The CM report is of the particular host where pyspark2 is running. Custom charts will provide the memory usage of Hadoop components only.
... View more
04-16-2018
02:08 AM
Hi, I have a cluster with 3 masters and 5 data nodes. Jupyter is installed on one of the master and runs via the following command in client mode pyspark2 --master yarn --deploy-mode client --name jupyter-notebook --driver-memory 4G --executor-memory 4G --executor-cores 3 --num-executers 3 Everything is working well but there is a difference of physical memory usage which is shown in Cloudera Manager and system memory usage Total physical memory : 62G Memory usage as shown by CM: 16.5 G Memory usage on the system: 56G Cloudera Manager while on the system CM only takes count of memory used by Hadoop components but not the yarn running in a client mode. Is there any way to make available the memory usage by yarn running in client-mode on CM? Thank you
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager