Member since
03-17-2016
132
Posts
106
Kudos Received
13
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1416 | 03-28-2019 11:16 AM | |
1687 | 03-28-2019 09:19 AM | |
1313 | 02-02-2017 07:52 AM | |
1736 | 10-03-2016 08:08 PM | |
627 | 09-13-2016 08:00 PM |
12-19-2019
06:26 AM
Please check if this file exists /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid . If not create the directory mkdir /var/run/hadoop-yarn/yarn/ chown -R yarn:hadoop /var/run/hadoop-yarn/yarn/ touch hadoop-yarn-nodemanager.pid chown yarn:hadoop /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid This will work.
... View more
04-02-2019
09:06 AM
You need to follow these as those are for spark thrift Configuring Cluster Dynamic Resource Allocation Manually To configure a cluster to run Spark jobs with dynamic resource allocation, complete the following steps: Add the following properties to the spark-defaults.conf file associated with your Spark installation (typically in the $SPARK_HOME/conf directory): Set spark.dynamicAllocation.enabled to true . Set spark.shuffle.service.enabled to true . (Optional) To specify a starting point and range for the number of executors, use the following properties: spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.minExecutors spark.dynamicAllocation.maxExecutors Note that initialExecutors must be greater than or equal to minExecutors , and less than or equal to maxExecutors . For a description of each property, see Dynamic Resource Allocation Properties. Start the shuffle service on each worker node in the cluster: In the yarn-site.xml file on each node, add spark_shuffle to yarn.nodemanager.aux-services , and then set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService . Review and, if necessary, edit spark.shuffle.service.* configuration settings. For more information, see the Apache Spark Shuffle Behavior documentation. Restart all NodeManagers in your cluster.
... View more
04-02-2019
08:52 AM
@Michael Bronson Can you paste output for hadoop fs -du -s -h /* and check hdfs configs for datanode data directory locations and check for the disk space of these directories specified for this property dfs.datanode.data.dir.
... View more
03-29-2019
06:37 AM
Have you followed these steps before adding nodelabels to yarn cluster ? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/configuring_node_labels.html
... View more
03-29-2019
06:18 AM
Follow this https://github.com/ehiggs/spark-terasort
... View more
03-29-2019
05:36 AM
@Anpan K It depends your environment. You can use a single AD with different Ou's created for different environment . Only thing you need to make sure is all the environments should be on same network if not so some natting would be require but still acheivable. .
... View more
03-29-2019
05:22 AM
Provide permission as 777 if you want all users to write into the folder
... View more
03-29-2019
04:43 AM
For this you need to use the spark dynamic allocation. Dynamic Allocation (of Executors) (aka Elastic Scaling) is a Spark feature that allows for adding or removing Spark executors dynamically to match the workload. Unlike the "traditional" static allocation where a Spark application reserves CPU and memory resources upfront (irrespective of how much it may eventually use), in dynamic allocation you get as much as needed and no more. It scales the number of executors up and down based on workload, i.e. idle executors are removed, and when there are pending tasks waiting for executors to be launched on, dynamic allocation requests them
... View more
03-29-2019
04:35 AM
In /etc/yum.repos.d, remove all .repo files pointing to the Internet and copy only .repo files from other servers which are already using your local repo. For HDP nodes, initially you need only 2 .repo files, one for the OS, and ambari.repo. When Ambari adds a new node to the cluster it will copy there HDP.repo and HDP-UTILS.repo. Also, have you set your repository URLs in Ambari-> Admin-> Stack and versions-> Versions -> Manage Versions -> [click on your current version] ?
... View more
03-28-2019
11:16 AM
1 Kudo
@Ruslan Fialkovsky You need to write a custom code which can block -skiptrash command This is the path where you need to place the command vi /usr/hdp/current/hadoop-client/bin/hadoop #!/bin/bash export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.6.5.0-292/hadoop} export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-/usr/hdp/2.6.5.0-292/hadoop-mapreduce} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-/usr/hdp/2.6.5.0-292/hadoop-yarn} export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec export HDP_VERSION=${HDP_VERSION:-2.6.5.0-292} export HADOOP_OPTS="${HADOOP_OPTS} -Dhdp.version=${HDP_VERSION}" exec /usr/hdp/2.6.5.0-292//hadoop/bin/hadoop.distro "$@" ###here you need to write code to restrict skip trash
... View more
03-28-2019
10:54 AM
You need to have a look at user limit factor for the queue and min max capacity https://community.hortonworks.com/content/supportkb/49640/what-does-the-user-limit-factor-do-when-used-in-ya.html https://hortonworks.com/blog/yarn-capacity-scheduler/
... View more
03-28-2019
09:19 AM
There's an API to remove older versions from the hosts. Take a look at https://issues.apache.org/jira/browse/AMBARI-18435 E.g., curl 'http://c6401.ambari.apache.org:8080/api/v1/clusters/cl1/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.5.0.0-1245"}}, "Requests/resource_filters": [{"hosts":"c6403.ambari.apache.org, c6402.ambari.apache.org"}]}'
... View more
03-28-2019
09:12 AM
@Michael Bronson Setting replication to 2 will be applicable only for files you create now. Use this command to decrease the hdfs size. hadoop dfs -setrep -w 2 -R /
... View more
03-27-2019
11:05 AM
It looks like Namenode is not able to reach datanodes. Please check for network issues and any recent changes made on hdfs configs like changing of rack etc.
... View more
04-24-2018
09:55 AM
Looks like logs are not completely aggregated to the application and there may be truncation of logs too .we can see from RM that for the application there is timeout for some of the nodemanagers . Job runs for 55 minutes and we have set yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds to 3600. Any Pointers.
... View more
Labels:
- Labels:
-
Apache YARN
02-12-2018
12:16 PM
1 Kudo
wget //ftp.icm.edu.pl/vol/rzm6/linux-slc/centos/7.1.1503/updates/x86_64/Packages/libtirpc-devel-0.2.4-0.8.el7_3.x86_64.rpm yum install libtirpc-devel-0.2.4-0.8.el7_3.x86_64.rpm -y This will resolve the issue. We have to do this on all the hosts
... View more
09-14-2017
05:16 PM
I have tried the destination as /tmp on hdfs and still faces the same issue
... View more
09-14-2017
11:53 AM
We are facing issue with insert ovewrite after upgrade from 2.5.3 to 2.6.1 Tried running with different destination . It created the folder but fails with below error Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/tmp/*] (state=42000,code=40000) Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/user/*] (state=42000,code=40000) Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
... View more
Labels:
- Labels:
-
Apache Hive
09-14-2017
11:50 AM
Hi Sindhu We are facing the same issue with insert ovewrite but it is not a local directory. We are facing this issue after upgrade from 2.5.3 to 2.6.1 Tried running with different destination . It created the folder but fails with below error Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/tmp/*] (state=42000,code=40000)
Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xyz] does not have [WRITE] privilege on [/user/*] (state=42000,code=40000)
Closing: 0: jdbc:hive2://host:2181,host:2181,host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
... View more
08-07-2017
01:28 PM
Have a look at this https://cwiki.apache.org/confluence/display/AMBARI/Using+APIs+to+delete+a+service+or+all+host+components+on+a+host
... View more
07-20-2017
09:32 AM
Have a look at this http://www.aosabook.org/en/hdfs.html
... View more
07-20-2017
09:32 AM
No, Namenode does this autometically
... View more
07-19-2017
07:55 AM
Try This https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_configuring_your_cluster_for_files_view.html
... View more
07-19-2017
07:52 AM
1 . First you need to run hadoop fsck / to check the under-replicated blocks. Then you can run hadoop -setrep 2 to the files which are under-replicated. This will stop the process. 2. Yes it will remove the third copy.
... View more
06-30-2017
04:23 PM
Thanks it help to resolve issue. I was missing with username principal
... View more
06-30-2017
01:45 PM
I have installed ranger earlier also and followed same steps, but my test connection is failing with HDFS repository
... View more
Labels:
- Labels:
-
Apache Ranger
03-14-2017
09:13 PM
Try this below URL jdbc:mysql://yourserver:3306/yourdatabase?zeroDateTimeBehavior=convertToNull
... View more