Member since
04-04-2017
10
Posts
1
Kudos Received
0
Solutions
04-10-2017
05:44 PM
Thanks Kshitij - will go down this route, as it seems to be a good fit.
... View more
04-06-2017
03:43 PM
1 Kudo
Currently on HDP 2.5, and looking at options for creating groups of users, based on experience, and limiting access to resources based these groups. For example beginner group users may get 10gb/4vcpu, intermediate gets 50gb/8vcpu, expert users get 200gb/16vcpu. The stack we are interested in is: Zeppelin -> Livy -> Spark -> YARN cluster Initially looking at doing this via YARN queues (i.e. user A can only submit jobs to the beginner queue), but from what I can tell, the Zeppelin Spark interpreter config only allows configuration of one queue, so regardless of user, all jobs would be sent to the same queue.
Another option would be to have multiple Zeppelin's, and configure different queues, but from what I can tell, if we have multiple Zeppelins all managed via Ambari, they will all get the same config. Am I off base? Is there a better way to limit resources (primarily memory) based on the user? Thanks!
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
-
Apache Zeppelin