Hi Esteban,   Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:   root.production root.adhoc root.users   ...you can define appropriate weights at amonst these 3 pools.   If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:   Weight  Pool 10         root.production 10         root.adhoc ...          root.[username] placement rule   In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.   Weight  Pool 10         root.production 10         root.adhoc 1           root.user1   Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:   Weight  Pool 10         root.production 10         root.adhoc 50          root.[username] generated pools (with 50 users)   So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:   Weight  Pool 10         root.production 10         root.adhoc 10         root.users              -> root.users.[username] placement rule   Thanks, Nick ... View more

Hi Maziyar and Li,   You are definitely on the right track here.  I see that the top-level root queue has "aclAdministerApps=maziyar,admin", which limits YARN API access to some of the time metrics for the application.  If you're not one of these users when you make the API call you won't get correct value returned for:   startedTime finishedTime elapsedTime logAggregationStatus amHostHttpAddress usedResources allocatedMB allocatedVCores runningContainers Although you will get some basic application info, so you'll see the application, but metrics will be wrong, just like you report.   So, the question is which user is the CM service using to interact with the YARN API.  I did some testing and reproduced your issue by limiting the aclAdministerApps property to "yarn".  I then found that when I add "dr.who" to aclAdministerApps at the root level, it starts working properly.   So, try modifying your root level ACLs to be "aclAdministerApps=maziyar,admin,dr.who", refresh the Dynamic Resource Pool (DRP) configuration, and see if it resolves the issue for you.   Nick ... View more

The console output is not stored anywhere besides in the CDSW Web UI itself.   If the user is executing YARN applications (Spark/Pyspark) then you would have the logs from any YARN containers stored as they normally would be and you could access them via the JobHistory Server Web UI.  Any Spark application run in CDSW will be in `--deploy-mode client` which means that the Spark Driver output is in the Web UI only - you have to manually copy it out of there and save in a file for future reference.   Nick ... View more

Hi Bhavesh,   I see that upstream Solr has added this capability via JIRA SOLR-4392.  This feature does not exist in the current CDH5.x but will almost certainly be available with CDH6 when that is released.   https://issues.apache.org/jira/browse/SOLR-4392   Nick ... View more