Cloudera Community
mike_bronson7
user rank
Guru

Re: HDFS is almost full 90% but data node disks ar...

by Guru in Support Questions
‎09-05-2018 12:11 PM
‎09-05-2018 12:11 PM
yes we know that , as you can see from all thread , the concoction is to add disks to each datanode , so no other solution exept this ... View more

Re: HDFS is almost full 90% but data node disks ar...

by Guru in Support Questions
‎09-05-2018 10:46 AM
‎09-05-2018 10:46 AM
remaining is only 18G ... View more

Re: what could be the cause for spark2-hdp-yarn-ar...

by Guru in Support Questions
‎09-05-2018 08:06 AM
‎09-05-2018 08:06 AM
@Jay . please let me know if I understand it as the following let say that one of the replica spark2-hdp-yarn-archive.tar.gz , is corrupted when I run this CLI su - hdfs -c "hdfs fsck /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz" dose its actually means that fsck will replace the bad one with the good one and status finally will be HEALTHY ? ... View more

Re: HDFS is almost full 90% but data node disks ar...

by Guru in Support Questions
‎09-05-2018 05:53 AM
‎09-05-2018 05:53 AM
@Jay , just to be sure , when you said "increase the DFS capacity" , you actually mean to add disks / add capacity by giving dfs.datanode.data.dir more mount points , am I right ... View more

Re: what could be the cause for spark2-hdp-yarn-ar...

by Guru in Support Questions
‎09-05-2018 05:33 AM
‎09-05-2018 05:33 AM
@Jay in spite this is diff case , I post yesterday the thred - https://community.hortonworks.com/questions/217423/spark-application-communicating-with-driver-in-hea.html , can you help me with this ? ... View more

Re: what could be the cause for spark2-hdp-yarn-ar...

by Guru in Support Questions
‎09-05-2018 05:29 AM
‎09-05-2018 05:29 AM
@Jay , very nice solution until now I was doing this , in ordeer to verify the file gzip -t /var/tmp/spark2-hdp-yarn-archive.tar.gz gunzip -c /var/tmp/spark2-hdp-yarn-archive.tar.gz | tar t > /dev/null tar tzvf spark2-hdp-yarn-archive.tar.gz > /dev/null ... View more

Re: HDFS is almost full 90% but data node disks ar...

by Guru in Support Questions
‎09-04-2018 08:27 PM
‎09-04-2018 08:27 PM
the replication factor is 3 , and we use all 4 disks for HDFS ( means 80G ) , regarding to what you said - how much hdfs space created , please advice how to check it? ... View more

spark application + communicating with driver in h...

by Guru in Support Questions
‎09-04-2018 08:24 PM
‎09-04-2018 08:24 PM
We run Spark App in Hadoop cluster ( HDP version – 2.6.4 , we have spark version 2.2 ) With the following details ExecutorInstances=6 Executor memory =35G spark2 daemon memory =50G from the yarn logs we can see many of these warning WARN executor.Executor: Issue communicating with driver in heartbeater What this messages means – “executor.Executor: Issue communicating with driver in heartbeater” And any ideas if needed additional spark configuration to solve this issue ? ... View more

Re: what could be the cause for spark2-hdp-yarn-ar...

by Guru in Support Questions
‎09-04-2018 07:33 PM
‎09-04-2018 07:33 PM
I cant tell you exaclty but after I tar again the files , this solve my problem ... View more

what could be the cause for spark2-hdp-yarn-archiv...

by Guru in Support Questions
‎09-04-2018 05:33 PM
‎09-04-2018 05:33 PM
hi all, we installed new hadoop cluster ( ambari + HDP version 2.6.4 ) after installation , we notice that we have problem with the spark-submit and finally we found that spark2-hdp-yarn-archive.tar.gz file is corruption full path - /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz ( from HDFS ) my question is - what could be the reason that this is is corrupted ? in spite this cluster is new fresh installation ... View more
Contact Me
Online
Offline
Last Visited
‎08-27-2024 09:17 AM
Community Statistics
Member Since ‎08-08-2017 09:40 AM
Last Visited ‎08-27-2024 09:17 AM
Posts 1,652
Kudos received 29