hello,   We are running cloudera hadoop 2.5.0-cdh5.3.0, on CentOS 6.5.   trying to get job preemption running, first submitting a long running spark job with spark-submit (spark 1.5.2) to a named queue. then submitting further sqoop jobs to the cluster to another named queue,   We are using Scheduling policy DRF, we have 3 queues, weighting 3:1:1, min share preemption timeout(seconds) is 60:300:300.   Yarn config is set with Admin ACL = * Fair Scheduler Preemption = true   Jobs are submitted and accepted by the RM; however no preemption is occurring - I am expecting to see the long running spark job interrupted and some resources diverted to the later submitted sqoop jobs.  instead all i am seeing is jobs accepted and queued up.   can anyone confirm this is the correct understanding and if so is there config i am missing?   thanks     ... View more

Hello   I'm having issues with the Python CM API, same as this post.   I am unable to install the host using a private key isntead of password:   cmd = cm.host_install(host_username, host_list, private_key="/home/ec2-user/.ssh/id_rsa", cm_repo_url=cm_repo_url)   2016-01-05 06:55:21,364 WARN NodeConfiguratorThread-21-0:com.cloudera.server.cmf.node.NodeConfigurator: Could not authenticate to ip-170-195-1-237.eu-west-1.compute.internal net.schmizz.sshj.common.SSHException: No provider available for Unknown key file   I have validated keys and can connect without password: [ec2-user@ip-170-195-1-237 cloudera-scm-server]$ ssh ec2-user@ip-170-195-1-237.eu-west-1.compute.internal Last login: Tue Jan  5 06:59:46 2016 from ip-170-195-1-237.eu-west-1.compute.internal [ec2-user@ip-170-195-1-237 ~]$   Pls advise   thanks   ... View more

Hello, I am testing queue accees control using ACLs for Yarn, I have the configuration as below: Hadoop 2.3.0-cdh5.1.0 CentOS release 6.4 (Final) yarn.acl.enable    true yarn.admin.acl    yarn Dynamic Resource Pools:     Root    submission: yarn    administration: anyone     default    submission: anyone    administration: Everyone is allowed to administer this pool because of inherited settings from the parent pools. [root@hostname ~]# mapred queue -info root 15/10/15 10:46:00 INFO client.RMProxy: Connecting to ResourceManager at hostname:8032 ====================== Queue Name : root Queue State : running Scheduling Info : Capacity: 0.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 You have new mail in /var/spool/mail/root [root@hostname ~]# mapred queue -showacls root.default 15/10/15 10:46:06 INFO client.RMProxy: Connecting to ResourceManager at hostname:8032 Queue acls for user :  root Queue  Operations ===================== root  ADMINISTER_QUEUE root.default  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS root.queue2  ADMINISTER_QUEUE root.queue1  ADMINISTER_QUEUE root.queue4  ADMINISTER_QUEUE root.queue3  ADMINISTER_QUEUE [root@hostname ~]# However when I run an application I am seeing the following warning and errors: client log: WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: Failed to run job : Application rejected by queue placement policy ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Failed to run job : Application rejected by queue placement policy resource manager log: resourcemanager.RMAuditLogger: USER=root     OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED       PERMISSIONS=Application rejected by queue placement policy      APPID=application_1444902104666_0001 resourcemanager.RMAppManager$ApplicationSummary: appId=application_1444902104666_0001,name=QueryResult.jar,user=root,queue=default,state=FAILED,trackingUrl=N/A,appMasterHost=N/A,startTime=1444902491251,finishTime=1444902491281,finalStatus=FAILED Can someone tell me pls why this is not working? from the config I understand root should be able to submit jobs to the default queue. Thanks Pete ... View more