Cloudera Community
smp_yuval
user rank
Contributor
My Accepted Solutions
Show All

Re: Memory allocation to executors in Spark

by Contributor in Support Questions
‎03-25-2018 10:04 AM
‎03-25-2018 10:04 AM
Thanks @Felix Albani. I didn't set swap at all for this test. (Removed swap from /etc/fstab and confirmed with 'free'). The settings of /proc/sys/vm/swappiness was 60 which is the default. I tried changing to 10 and 0 as you suggested, but still, I don't see that more RAM is used. Any other idea? ... View more

Memory allocation to executors in Spark

by Contributor in Support Questions
‎03-21-2018 06:13 PM
‎03-21-2018 06:13 PM
I'm running a Spark application under yarn cluster. I have 250GB available in my one-node cluster and try to fully utilize it. I've set for each executor the following: spark.executor.memory 28672M (= 28G ) spark.yarn.executor.memoryOverhead 2048 (approx 7%) I expected to see by monitoring with "top" that each executor is utilizing the allocated memory. However, I found that the resident memory is use is ~10GB and the virtual memory is ~30GB. The yarn log (/var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-host.log) say the same: 2018-03-21 15:09:49,838 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 279993 for container-id container_e09_1521632768675_0005_01_000007: 11.0 GB of 32 GB physical memory used; 31.5 GB of 67.2 GB virtual memory used I repeated few times with various spark.executor.memory settings. In all my tests, the resident memory in use was less than 40% of my settings, as in the example above. How can I utilize the entire 32GB that I've allocated ?? ... View more
Labels:

Re: How to use config.py

by Contributor in Support Questions
‎02-19-2018 05:47 PM
‎02-19-2018 05:47 PM
Thanks @Jay Kumar SenSharma. Playing with your reply I learned that "--host" is mandatory. In addition, I figured that I had a typo in one of the parameters (-e instead of -c) so it is better to write out the long format and not the abriviation... ... View more

How to use config.py

by Contributor in Support Questions
‎02-15-2018 10:01 PM
1 Kudo
‎02-15-2018 10:01 PM
1 Kudo
I'm trying to read (and later to set) some of hadoop configuration using configs.py but it fails and I'm unable to figure out what is wrong. This is what I tried: [root@hdptst55 ~]# /var/lib/ambari-server/resources/scripts/configs.py -a get -n host_group_1 -u admin -p admin -e yarn-site.xml -k yarn.nodemanager.resource.memory-mb 2018-02-15 16:51:43,771 ERROR File yarn-site.xml doesn't exist or you don't have permissions. [root@hdptst55 ~]# The file and the parameters do exists: [root@hdptst55 ~]# grep -a1 yarn.nodemanager.resource.memory-mb /etc/hadoop/conf/yarn-site.xml <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>5120</value> [root@hdptst55 ~]# What is wrong? Are there logs to this tool? Were can I find some practical usage examples? ... View more

Re: Need help installing HDP 2.6

by Contributor in Support Questions
‎01-24-2018 05:04 PM
‎01-24-2018 05:04 PM
in addition to installing mysql-connector-java-5.1.45-bin, I added a link to it: ln -s /usr/lib/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar I already disable/stop firewalld as in the document. In addition, I stopped iptables (using: iptables -F). This is not in the doc. not sure if this helped or not. I now have working environment. Thank you @Jay Kumar SenSharma & @Shyam Sunder Rai for your valuable comments. ... View more

Re: Need help installing HDP 2.6

by Contributor in Support Questions
‎01-23-2018 08:27 PM
‎01-23-2018 08:27 PM
Thanks @Shyam Sunder Rai. Before starting the install, I followed the instructions and: systemctl disable firewalld service firewalld stop setenforce 0 In addition I uninstalled PackageKit. Anything else that I had to do ? ... View more

Re: Need help installing HDP 2.6

by Contributor in Support Questions
‎01-23-2018 04:20 PM
‎01-23-2018 04:20 PM
Attaching STDOUT + STDERR stderr95txt.txt stdout95txt.txt ... View more

Need help installing HDP 2.6

by Contributor in Support Questions
‎01-23-2018 04:12 PM
‎01-23-2018 04:12 PM
Dear community, I have very hard time installing HDP 2.6.4 on Centos 7.4. Would appreciate help. I followed carefully the instructions at https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/ch_Getting_Ready.html Before starting the wizard: I installed & started MySql (using the instructions here: https://linode.com/docs/databases/mysql/how-to-install-mysql-on-centos-7/). Installed (=extracted) mysql-connector-java-5.1.45.tar.gz to /usr/lib/ run: ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/lib/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar After starting the server, I followed the the wizard. At the "Install-start-test, "Hive Metastore Start" failes. The error it fails on is: http://hdptst5:8080/resources//mysql-connector-java.jar due to HTTP error: HTTP Error 404: Not Found (Attached is the entire log) Any help would be appreciated. --yuval ... View more
Contact Me
Online
Offline
Last Visited
‎04-09-2018 06:30 PM
Community Statistics
Member Since ‎01-23-2018 03:13 PM
Last Visited ‎04-09-2018 06:30 PM
Posts 13
Kudos received 1