If I wanted to get a very powerful PC running 32mb or 64mb RAM, that I could be running the VM HDP Sandbox on, if this possible to have the sandbox use all that memory? I would like to use it to ssh into it remotely and run some basic queries.
This would be more for personal use than anything, but could expand. Is this a good idea? Some of the datasets I could be running would be greater than 1mil rows.
You can increase the memory for the Virtual Machine by changing the settings in VirtualBox.
1. Open VirtualBox.
2. Select the VM.
3. Right click and select Settings.
4. Select the System tab and increase the Base Memory setting:
I have a 32 GB Windows desktop and run the Sandbox with 16GB of memory and it works well. However, I have not tried using the sandbox with that large a data set. Another option for spinning up a test cluster is to use Hortonworks Data Cloud to create a cluster on AWS. Another option is Azure. See this article for more information:
A third option is open Ambari, shut down any services you are not using and put the down services in maintenance mode.
@josh janzen It depends on how long you will need the cluster. If you use SSD storage you can bring your AWS cluster down when you don't need it. You will have to pay for the storage if using SSD. There are other options where you can use ephemeral clusters and store your data in S3.