@saivenkatg55Can you please  share the hdfs-site.xml ... View more

Please check if this file exists /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid . If not create the directory   mkdir /var/run/hadoop-yarn/yarn/ chown -R yarn:hadoop /var/run/hadoop-yarn/yarn/ touch hadoop-yarn-nodemanager.pid chown yarn:hadoop /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid   This will work.     ... View more

You need to follow these as those are for spark thrift Configuring Cluster Dynamic Resource Allocation Manually To configure a cluster to run Spark jobs with dynamic resource allocation, complete the following steps: Add the following properties to the spark-defaults.conf file associated with your Spark installation (typically in the $SPARK_HOME/conf directory): Set spark.dynamicAllocation.enabled to true . Set spark.shuffle.service.enabled to true . (Optional) To specify a starting point and range for the number of executors, use the following properties: spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.minExecutors spark.dynamicAllocation.maxExecutors Note that initialExecutors must be greater than or equal to minExecutors , and less than or equal to maxExecutors . For a description of each property, see Dynamic Resource Allocation Properties. Start the shuffle service on each worker node in the cluster: In the yarn-site.xml file on each node, add spark_shuffle to yarn.nodemanager.aux-services , and then set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService . Review and, if necessary, edit spark.shuffle.service.* configuration settings. For more information, see the Apache Spark Shuffle Behavior documentation. Restart all NodeManagers in your cluster. ... View more

@Michael Bronson Can you paste output for hadoop fs -du -s -h /* and check hdfs configs for datanode data directory locations and check for the disk space of these directories specified for this property dfs.datanode.data.dir. ... View more

Have you followed these steps before adding nodelabels to yarn cluster ? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/configuring_node_labels.html ... View more

Follow this https://github.com/ehiggs/spark-terasort ... View more

@Anpan K It depends your environment. You can use a single AD with different Ou's created for different environment . Only thing you need to make sure is all the environments should be on same network if not so some natting would be require but still acheivable. . ... View more

Provide permission as 777 if you want all users to write into the folder ... View more

For this you need to use the spark dynamic allocation. Dynamic Allocation (of Executors) (aka Elastic Scaling) is a Spark feature that allows for adding or removing Spark executors dynamically to match the workload. Unlike the "traditional" static allocation where a Spark application reserves CPU and memory resources upfront (irrespective of how much it may eventually use), in dynamic allocation you get as much as needed and no more. It scales the number of executors up and down based on workload, i.e. idle executors are removed, and when there are pending tasks waiting for executors to be launched on, dynamic allocation requests them ... View more

In /etc/yum.repos.d, remove all .repo files pointing to the Internet and copy only .repo files from other servers which are already using your local repo. For HDP nodes, initially you need only 2 .repo files, one for the OS, and ambari.repo. When Ambari adds a new node to the cluster it will copy there HDP.repo and HDP-UTILS.repo. Also, have you set your repository URLs in Ambari-> Admin-> Stack and versions-> Versions -> Manage Versions -> [click on your current version] ? ... View more

You need to have a look at user limit factor for the queue and min max capacity https://community.hortonworks.com/content/supportkb/49640/what-does-the-user-limit-factor-do-when-used-in-ya.html https://hortonworks.com/blog/yarn-capacity-scheduler/ ... View more

@Michael Bronson Setting replication to 2 will be applicable only for files you create now. Use this command to decrease the hdfs size. hadoop dfs -setrep -w 2 -R / ... View more

It looks like Namenode is not able to reach datanodes. Please check for network issues and any recent changes made on hdfs configs like changing of rack etc. ... View more

wget //ftp.icm.edu.pl/vol/rzm6/linux-slc/centos/7.1.1503/updates/x86_64/Packages/libtirpc-devel-0.2.4-0.8.el7_3.x86_64.rpm yum install libtirpc-devel-0.2.4-0.8.el7_3.x86_64.rpm -y This will resolve the issue. We have to do this on all the hosts ... View more

I have tried the destination as /tmp on hdfs and still faces the same issue ... View more

Hi Sindhu We are facing the same issue with insert ovewrite but it is not a local directory. We are facing this issue after upgrade from 2.5.3 to 2.6.1 Tried running with different destination . It created the folder but fails with below error Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/tmp/*] (state=42000,code=40000) Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/user/*] (state=42000,code=40000) Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 ... View more

No views don't support that ... View more

You can configure two separate hive views for two servers ... View more

Have a look at this https://cwiki.apache.org/confluence/display/AMBARI/Using+APIs+to+delete+a+service+or+all+host+components+on+a+host ... View more

Have a look at this http://www.aosabook.org/en/hdfs.html ... View more

No, Namenode does this autometically ... View more

Try This https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_configuring_your_cluster_for_files_view.html ... View more

1 . First you need to run hadoop fsck / to check the under-replicated blocks. Then you can run hadoop -setrep 2 to the files which are under-replicated. This will stop the process. 2. Yes it will remove the third copy. ... View more

Thanks it help to resolve issue. I was missing with username principal ... View more

Try this below URL jdbc:mysql://yourserver:3306/yourdatabase?zeroDateTimeBehavior=convertToNull ... View more