Hi @rushi_ns ,   yours might be completly different issue. Please create a new Question thread stating your issue.   ... View more

@Constantin Stanca There was no error in hive/hdfs/yarn etc. during that time frame, even I tried to restart all the services but didn't helped. But when I restarted Ambari server the issue got resolved, don't know how this exception is related to ambari restart 🙂 ... View more

I have increased the heap size to 1638 MB , Metastore heap size to 8052 Mb , Client heap size to 1024 mb but still i am getting this random behavior . Please suggest. ... View more

It worked for me. Thanks! ... View more

Now you are getting same error for host cmilpoclab2.amdocs.com, Please check if you have mentioned correct ssh key while adding nodes on Ambari UI and all nodes have correct /etc/hosts entries. If nothing works then try removing all HDP packages from host cmilpoclab2.amdocs.com and register it again. ... View more

@Sagar Shimpi Please find the output of the commands below: [root@RHEL01 ~]# hostname RHEL01 [root@RHEL01 ~]# hostname -f RHEL01.ind.hp.com [root@RHEL01 ~]# cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 [root@RHEL01 ~]# ifconfig eth0 Link encap:Ethernet HWaddr FA:16:3E:D3:E3:4D inet addr:192.168.52.104 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fed3:e34d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1400 Metric:1 RX packets:495960 errors:0 dropped:0 overruns:0 frame:0 TX packets:501687 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:359851713 (343.1 MiB) TX bytes:585067235 (557.9 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:728770 errors:0 dropped:0 overruns:0 frame:0 TX packets:728770 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:915678123 (873.2 MiB) TX bytes:915678123 (873.2 MiB) [root@RHEL01 ~]# netstat -nlp --inet Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1038/rpcbind tcp 0 0 0.0.0.0:4242 0.0.0.0:* LISTEN 4560/jsvc.exec tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1303/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1088/cupsd tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1530/postmaster tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1321/sendmail tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 4249/java tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 4249/java tcp 0 0 0.0.0.0:35227 0.0.0.0:* LISTEN 1056/rpc.statd tcp 0 0 0.0.0.0:8670 0.0.0.0:* LISTEN 2572/python tcp 0 0 0.0.0.0:50079 0.0.0.0:* LISTEN 4560/jsvc.exec tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN 4560/jsvc.exec tcp 0 0 127.0.0.1:56648 0.0.0.0:* LISTEN 4249/java tcp 0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 4249/java udp 0 0 0.0.0.0:111 0.0.0.0:* 1038/rpcbind udp 0 0 0.0.0.0:631 0.0.0.0:* 1088/cupsd udp 0 0 0.0.0.0:47113 0.0.0.0:* 1056/rpc.statd udp 0 0 0.0.0.0:4242 0.0.0.0:* 4560/jsvc.exec udp 0 0 0.0.0.0:789 0.0.0.0:* 1038/rpcbind udp 0 0 127.0.0.1:40 0.0.0.0:* 4560/jsvc.exec udp 0 0 0.0.0.0:808 0.0.0.0:* 1056/rpc.statd udp 0 0 0.0.0.0:68 0.0.0.0:* 959/dhclient [root@RHEL01 ~]# ls -lR /var/run/hadoop/ /var/run/hadoop/: total 16 drwxrwxr-x. 2 hdfs hadoop 4096 Jun 28 03:55 hdfs drwxrwxr-x. 2 mapred hadoop 4096 Jun 27 08:18 mapreduce drwxr-xr-x. 2 root root 4096 Jun 28 05:04 root drwxrwxr-x. 2 yarn hadoop 4096 Jun 27 08:18 yarn /var/run/hadoop/hdfs: total 8 -rw-r--r-- 1 hdfs hadoop 5 Jun 28 03:55 hadoop-hdfs-datanode.pid -rw-r--r-- 1 hdfs hadoop 5 Jun 28 03:55 hadoop-hdfs-namenode.pid /var/run/hadoop/mapreduce: total 0 /var/run/hadoop/root: total 12 -rw-r--r-- 1 root root 5 Jun 28 03:55 hadoop-hdfs-nfs3.pid -rw------- 1 root root 5 Jun 28 03:55 hadoop_privileged_nfs3.pid -rw-r--r-- 1 root root 6 Jun 28 05:04 hadoop-root-namenode.pid /var/run/hadoop/yarn: total 0 [root@RHEL01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 16.181.235.174 RHEL01.ind.hp.com RHEL01 16.181.235.175 RHEL02.ind.hp.com RHEL02 16.181.235.176 RHEL03.ind.hp.com RHEL03 [root@RHEL01 ~]# ... View more

This is going to be one of two issues: Disk I/O or Executors. With count() you will not be doing any swapping. Set the number of partitions executors to (cores * nodes) - 2. Let's assume 8 cores, then that's 30 for you or val rdd = sc.textFile("some file",30) That being said, I don't see how a shuffle is going to help a simple count. That is going to execute on each partition without shuffle and return to the driver. You can perform a test by changing count() to saveAsTextFile(), but I'm thinking you are bound by disk I/O. Are you in a cloud environment? Try to do a reduceByKey() then count(). If your processing time is still about the same, then it further points to disk I/O. ... View more

in the documentation http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_software_requirements.html you have minimum requirements : OpenSSL (v1.01, build 16 or later) ... View more

This question is old, but I am having trouble with this as well. Using apache zeppelin 0.8.0. The dependency is: com.typesafe.play:play-json_2.11:2.6.10 util.Properties.versionString indicated the Scala version was 2.11 Unfortunately, whenever this dependency is present, I cannot define a case class: case-class-error.txt ... View more

Hi, My .csv file doesn't have any such separators. Verified in notepad. Still I'm facing the issue. Any help? ... View more

Hello, have you tried running any other VM to see if virtualbox is installed correctly? If that's the case, please check this post about installing virtualbox in centos7, it sounds like you are missing the epel dkms package: http://tecadmin.net/install-oracle-virtualbox-on-centos-redhat-and-fedora/# ... View more

Hello @emad rifai Could you please share how did you resolve issue with org.apache.spark.SparkException: Yarn application has already ended! Thanks in advance, Olga ... View more

Hi! I know it's a fairly old post / question but google keeps bringing this up as one of the top search results in this topic. I've managed to find a workaround for the GUI issue in HDP 2.6.5+, and I've also created a step-by-step guide to do so. It's available at https://blog.hegistvan.net/2018/08/how-to-install-gnome-on-top-of-hdp.html Hope it helps, Hegi ... View more