- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how to increase spark's worker memory from CM
- Labels:
-
Apache Spark
-
Cloudera Manager
Created on ‎03-12-2015 01:37 AM - edited ‎09-16-2022 02:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i am working on a cluster of 1 master and 2 workers , when i open spark master UI
Memory: 128.0 MB Total
each worker memory 64.0 MB (0.0 B Used)
while each worker has 4 GB memory in total and 2.5 GB free memory , so i want to increase the worker memory instead of 64 to make it 1 GB , i tried to find spark-defaults.conf in /etc/spark/conf.cloudera.spark but it doesn't exist , i am running cloudera manager 5.0.2 , i guess there might be a configuration that can be done to increase the memory from cloudera manger => spark service , but i can't find it
Thanks in advance
Created ‎03-12-2015 08:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can also search for "memory" or "heap" using the search bar on the left.
Editing /etc/spark/conf won't help you since the daemon roles don't read that file. CM daemons don't read from /etc, but instead have their own independent configuration directories.
http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/
Thanks,
Darren
Created ‎03-12-2015 08:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can also search for "memory" or "heap" using the search bar on the left.
Editing /etc/spark/conf won't help you since the daemon roles don't read that file. CM daemons don't read from /etc, but instead have their own independent configuration directories.
http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/
Thanks,
Darren
Created ‎03-12-2015 09:27 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks so much !!
