question Re: Number of Concurrent Users on HDP Sandbox in a Classroom? in Support Questions

Number of Concurrent Users on HDP Sandbox in a Classroom?

c-zakkaroff — Tue, 21 May 2019 10:41:25 GMT

Hello everyone.

I would really appreciate some help/advice with this.

I run HDP Sandbox 3.1 on in a VM with 32GB memory, and plenty of disk space, for educational purposes. However, as soon as there more than two or three logged in users running Hive queries, the jobs get stuck in the queue, with one job listed as "Running", and everything else as "Accepted". And those are not even big jobs -- simply some exercises from HDP Sandbox tutorials.

Can anyone suggest, is it possible to run HDP Sandbox with 25 to 50 concurrent users, working on some basic exercises from HDP Sandbox tutorials? What would I need to change in the configs for that?

Many thanks!

Re: Number of Concurrent Users on HDP Sandbox in a Classroom?

gdeleon — Wed, 22 May 2019 08:08:13 GMT

1. The sandbox is not meant to be used by multiple users.
2. Take a look at Appendix B: Troubleshoot in Learning the Ropes of the HDP Sandbox

Re: Number of Concurrent Users on HDP Sandbox in a Classroom?

LesterMartin — Wed, 22 May 2019 09:21:09 GMT

As @gdeleon suggested... "that dog won't hunt". Basically, you'll need at least two YARN containers for each Hive user/query going on to house the applicationMaster and another container to start doing some actual work (the first one there is getting their application into the "Running" state). The "Accepted" state means those users were able to get a container for their applicationMasters, but then there isn't enough space for YARN to grant enough actual containers to do much else. Again, it is just isn't designed for this.

A better solution would be to let each student have their own HDP Sandbox (and the won't need to allocate 32GB VMs). Good luck and happy Hadooping!

Re: Number of Concurrent Users on HDP Sandbox in a Classroom?

c-zakkaroff — Tue, 16 Jul 2019 06:51:27 GMT

Thank you very much, @Lester Martin and @gdeleon, for the explanations and suggestions.

I've tried the HDP Sandbox per student solution, and while it works, there's a bit of overhead with VM management.

I'm wondering though, is there another path to a minimum multi-user Hadoop environment for a class environment, whether it's using HDP Sandbox or something of this sort. It is great to have Ambari and everything else that comes with HDP Sandbox.