Member since
08-10-2017
108
Posts
2
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2901 | 01-28-2019 08:41 AM | |
4818 | 01-28-2019 08:35 AM | |
2656 | 12-18-2018 05:42 AM | |
7813 | 08-16-2018 12:12 PM | |
3008 | 07-24-2018 06:55 AM |
08-12-2018
04:37 AM
@Sandeep Nemuri @Arti Wadhwani Please suggest.
... View more
08-12-2018
02:46 AM
@Mohammad Hamdan Here is content of capacity scheduler. Please suggest: yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=25
yarn.scheduler.capacity.root.default.maximum-capacity=50
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=2
yarn.scheduler.capacity.root.queues=default,hs2,longrun,sqoop,talend
yarn.scheduler.capacity.queue-mappings=u:talenduser:talend,u:dbenefi:longrun,u:sqoop:sqoop
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.hs2.acl_submit_applications=*
yarn.scheduler.capacity.root.hs2.capacity=25
yarn.scheduler.capacity.root.hs2.maximum-am-resource-percent=0.4
yarn.scheduler.capacity.root.hs2.maximum-applications=500
yarn.scheduler.capacity.root.hs2.maximum-capacity=100
yarn.scheduler.capacity.root.hs2.ordering-policy=fair
yarn.scheduler.capacity.root.hs2.ordering-policy.fair.enable-size-based-weight=true
yarn.scheduler.capacity.root.hs2.state=RUNNING
yarn.scheduler.capacity.root.hs2.user-limit-factor=4
yarn.scheduler.capacity.root.longrun.acl_submit_applications=*
yarn.scheduler.capacity.root.longrun.capacity=15
yarn.scheduler.capacity.root.longrun.maximum-capacity=15
yarn.scheduler.capacity.root.longrun.state=RUNNING
yarn.scheduler.capacity.root.longrun.user-limit-factor=1
yarn.scheduler.capacity.root.sqoop.acl_submit_applications=*
yarn.scheduler.capacity.root.sqoop.capacity=10
yarn.scheduler.capacity.root.sqoop.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.sqoop.maximum-capacity=100
yarn.scheduler.capacity.root.sqoop.state=RUNNING
yarn.scheduler.capacity.root.sqoop.user-limit-factor=10
yarn.scheduler.capacity.root.talend.acl_submit_applications=*
yarn.scheduler.capacity.root.talend.capacity=25
yarn.scheduler.capacity.root.talend.maximum-am-resource-percent=0.4
yarn.scheduler.capacity.root.talend.maximum-applications=500
yarn.scheduler.capacity.root.talend.maximum-capacity=100
yarn.scheduler.capacity.root.talend.ordering-policy=fair
yarn.scheduler.capacity.root.talend.ordering-policy.fair.enable-size-based-weight=true
yarn.scheduler.capacity.root.talend.state=RUNNING
yarn.scheduler.capacity.root.talend.user-limit-factor=4
... View more
08-11-2018
03:56 AM
amshare value is 0.4 i.e. 40%. Whats the meaning of -1.0f? I did not get.
... View more
08-11-2018
03:20 AM
Please find attached RM UI and RM Scheduler screenshou. Will attach RM logs soon. Please suggest.rm-scheduler.png rm-ui.png
... View more
08-11-2018
02:13 AM
@Mohammad Hamdan Min container size is 4GB and max container size is 8GB.
... View more
08-10-2018
01:49 PM
Hello All, We are using HDP-2.5.6 in Production cluster. We have 5 queues configured in capacity scheduler. We have given 840 GB of total memory to Yarn. Preemption is enabled for all queue. Out of that for 'talend' queue we have given 25% as min capacity and 100% as max capacity. Attached screenshot for 'talend' queue talend-queue.png. The problem is, though 'talend' queue has lot of resources available applications are going into accepted state for long time. Attached screenshot for accepted application accepted-application.png. Some applications are even in accepted state for more that 5 hours. Diagnostic section is showing wrong AM resource. In ResourceManager logs, we are getting following message: 2018-08-10 06:12:13,129 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(662)) - Not activating application application_1529949441873_131872 as amIfStarted: <memory:348160, vCores:85> exceeds amLimit: <memory:344064, vCores:1>
2018-08-10 06:12:13,129 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(662)) - Not activating application application_1529949441873_131873 as amIfStarted: <memory:348160, vCores:85> exceeds amLimit: <memory:344064, vCores:1>
2018-08-10 06:12:13,129 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(662)) - Not activating application application_1529949441873_131874 as amIfStarted: <memory:348160, vCores:85> exceeds amLimit: <memory:344064, vCores:1>
We have AM resources available in queue but it shows exceeds amLimit message. Also amLimit is not correct. Why applications are going in accepted state though we have lot of resources available at queue level? Please suggest.
... View more
Labels:
- Labels:
-
Apache YARN
08-02-2018
07:09 AM
@Vipin Rathor I am logging in using bhushan-kandalkar user but in Zeppelin Notebook UI, I am still getting 'bhushan-kandalkar@test.com' user for every notebook permission.
... View more
08-01-2018
12:49 PM
@Felix Albani @Vipin Rathor I performed all these steps. I can see that user 'bhushan-kandalkar' is able to view notebooks created by 'bhushan-kandalkar@test.com' user. That's good news. But in Zeppelin Notebook UI, I am still getting 'bhushan-kandalkar@test.com' user. Attached screenshot notebook-permission.png. I am not getting ''bhushan-kandalkar' user. Also, when I share notebook to other user, that notebook is removed from the user who has shared that notebook. That should not happen. Please suggest.
... View more
07-31-2018
05:15 PM
Thanks @Felix Albani I already did that. But note permission for every notebook is still showing 'bhushan-kandalkar@test.com' as owner, reader and writer. Also, when I tried to change owner name to 'bhushan-kandalkar' its giving permission denied error. Please suggest.
... View more
07-31-2018
09:40 AM
Hello All, We have synced AD users in Zeppelin using shiro.ini file. User is able to login using entire domain name eg: 'bhushan-kandalkar@test.com' All notebook have owners set to 'bhushan-kandalkar@test.com' Now, we have synced AD users in Zeppelin using SaMAcoountName. Hence User is able to login using SaMAcoountName only eg: 'bhushan-kandalkar' But problem is user 'bhushan-kandalkar' is not able to view notebooks created by 'bhushan-kandalkar@test.com' though its an same user. How should I change owners of all notebooks from 'bhushan-kandalkar@test.com' to 'bhushan-kandalkar'? Please suggest. Thanks, Bhushan
... View more
Labels:
- Labels:
-
Apache Zeppelin