Member since
02-10-2019
47
Posts
9
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3856 | 07-15-2019 12:04 PM | |
2984 | 11-03-2018 05:00 AM | |
5285 | 10-24-2018 07:38 AM | |
6138 | 10-08-2018 09:47 AM | |
1590 | 08-17-2018 06:33 AM |
07-02-2018
05:47 AM
Can click on the application in RM UI and see what is reported in Diagnostics and paste the content ? It should specify the reason why the job is still in accepted state.
... View more
06-29-2018
03:01 PM
An easy way to check the maximum am resource is in RM UI for queue q4 http://rm-host:8088/cluster/scheduler?openQueues=Queue:%20q4 Check the values for Max Application Master Resources and Used Application Master Resources . Also you can check other values here which will be useful to identify your queue limits configured.
... View more
06-29-2018
02:43 PM
There are additional parameters which limit the usage of queue resources by a single user or the application master. They are yarn.scheduler.capacity.<queue-path>.user-limit-factor and yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent . The detailed documentation of these capacity-scheduler properties can be referred at http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Properties Property Default Behaviour and Recommendation yarn.scheduler.capacity.<queue-path>.user-limit-factor 1 If you are submitting jobs as the same user, it is recommended to increase the value above 1. Otherwise the same user can't submit more than one job which exceeds the queue capacity. For q4, a single user can only utilize the max 100% if this is set to 10. This is likely the reason the new job is not getting executed. yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent 0.1 For example if you consider the default value, in q4 only 10% of the max-capacity can be used. When multiple applications are launched to the same queue, then new applications wont be accepted even if resources are free in the cluster
... View more
05-03-2018
10:24 AM
I didn't notice that you were only setting YARN_RESOURCEMANAGER_OPTS. This env variable is used for only the resourcemanger daemon. So to specify the opts for all hadoop and yarn client commands, you can use HADOOP_CLIENT_OPTS in . hadoop-env.sh . export HADOOP_CLIENT_OPTS="-Dyarn.resourcemanager.hostname=192.168.33.33" But I am not sure why you would need to this when you can just set it in the yarn-site.xml, which is what is recommended.
... View more
05-01-2018
06:33 AM
yarn-env.sh is used when you run any yarn command. So it works if you use the yarn command to submit a mapreduce job as below. yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 5 5 But spark-submit command doesn't invoke yarn-env.sh, so it will read the yarn-site.xml from $HADOOP_CONF_DIR and gets resourcemanager address from it.
... View more
04-18-2018
07:17 AM
@Purna Chandra Mahesh Bhogavalli There is no sort or order_by query parameter supported in the ResourceManager rest api. For your curl request, you can check the documentation at http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API The only option now is to sort at the client side. For command line, one easy way is to install a tool called jq https://stedolan.github.io/jq/download/ if you are interested. For your use case, you can do something like follows: curl -get "http://resource-manager-hostname:8088/ws/v1/cluster/apps?state=running&limit=20" | jq '.apps.app|sort_by(.queueUsagePercentage)' For descending sort you can use, curl -get "http://resource-manager-hostname:8088/ws/v1/cluster/apps?state=running&limit=20" | jq '.apps.app|sort_by(.queueUsagePercentage)|reverse' Hope this helps.
... View more
04-16-2018
11:40 AM
Currently in hadoop only simple authentication and kerberos (SPNEGO) authentication is supported out of the box. So your requirement is not supported. Refer http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/HttpAuthentication.html
... View more
04-16-2018
11:26 AM
@ASIF Khan Hadoop’s Pseudo/Simple authentication can be used if the cluster is not kerberized. The user name must be specified in the first browser interaction using the user.name query string parameter. e.g. http://rm-host:8088/cluster?user.name=asif.
... View more
04-16-2018
09:41 AM
Was that the entire content? Can you upload the file yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.log?
... View more
04-16-2018
09:27 AM
Do you see any errors in Resourcemanager logs? They are usually at /var/log/hadoop-yarn/yarn/ . Do you have ResourceManager HA enabled?
... View more
- « Previous
- Next »