Member since
10-26-2016
98
Posts
11
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2181 | 09-16-2019 04:00 AM | |
1588 | 04-21-2017 01:43 PM | |
1272 | 03-30-2017 11:35 AM | |
4544 | 12-26-2016 10:19 AM |
01-06-2020
11:03 PM
@Shelton Sorry for late response. I followed the thread and was able to configure Fair Scheduler successfully. Thank you for your help. Cheers.
... View more
01-02-2020
09:30 PM
Hi @Shelton Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.
... View more
01-02-2020
08:54 AM
Hi Team, I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-hortonworks-data-platform it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error. 2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml
2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)
2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512) Any help on this issue.
... View more
Labels:
11-18-2019
03:50 AM
@sagarshimpi Thank you.
... View more
11-18-2019
02:39 AM
@sagarshimpi Thank you for the link. Is it okay to remove the PIG clients if there no such use cases. Will this impact any of the service in the cluster.
... View more
11-18-2019
12:45 AM
Hi Team,
Could you please help me to understand why installing Hive Service requires PIG clients to be installed. Also, is it okay to remove PIG clients after installing hive.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
10-15-2019
06:59 AM
Hive's execution engine supports MapReduce & Tez. Does it supports Spark in HDP 3.x.
https://issues.apache.org/jira/browse/HIVE-7292
... View more
Labels:
09-16-2019
04:00 AM
I was able to figure out with the help of below community article. https://community.cloudera.com/t5/Community-Articles/How-to-install-all-clients-on-all-nodes-in-your- cluster/ta-p/246821 script.sh
---------
curl -s -u admin:admin http://ambari:8080/api/v1/hosts|grep host_name|sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p'>hostcluster.txt
while read line ;
do
j=$line
mkdir -p $j
curl -s -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo":{"context":"Restart HDFS Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://ambari:8080/api/v1/clusters/rest/hosts/$j/host_components/HDFS_CLIENT
done < hostcluster.txt
... View more
09-16-2019
01:40 AM
Below article shows how to restart clients on Specific host. https://cwiki.apache.org/confluence/display/AMBARI/Restarting+host+components+via+the+API Is there any way we can perform the same on all the hosts in the cluster at one time. Thank you in advance.
... View more
Labels:
- Labels:
-
Ambari Blueprints
-
Apache Ambari