increasing VDI size doesn't mean partitions/filesystems inside Guest VM will be increased - I figured this out. Anyway, I have been able to successfully increase the capacity of HDFS. I'm facing problems loading a 73GB file into hdfs. See my most recent post for more details (before this). ... View more

@aakulov @Shelton Using the hdfs dfs -put has given me the following error. Could it be because my virtual machine (and the hadoop cluster running on top of it), the VDI file all reside on a external hard drive? It is not an SSD and read/write speeds aren't extremely fast. I believe I can get better performance if everything ran off my machine's own internal hard drive (which is an SSD).    Would using distcp help in this scenario?   ... View more

Thank you for your inputs. I have been able to expand the size of my HDFS finally.  ... View more

@Shelton  So, I've been able to create a new partition and format it as an ext4 filesystem mount it How do I add add this new partition to my datanode? Is it as simple as putting the drive path in Amabari DataNode config? ... View more

Initially, I was unable to load data into my guest OS and therefore I used Amabari's GUI to load data into HDFS directly from my host OS. Now that I have figured out using WinSCP, I have been able to copy my file from the host OS to the guest OS. I have used the put command to copy files to my HDFS. However, I have run out of disk space on my virtual machine. I have increased my VDI size but still figuring out a way to ensure HDFS utilizes the newly allocated storage. Once I figure that out, I think loading the data won't be a problem anymore. I did read the documentation of distcp yesterday. Thank you for giving a summary here anyway. I may not be required to use distcp. Let me see how it goes. ... View more

It is a VDI. I have used Virtual Media Manager to increase the size of my disk. How can i get HDFS to expand and make use of the unallocated space?   I'm assuming this is how one would do it 1. Create a new partition in the Guest OS and assign a mount point to it. 2. Add that path to the DataNode directories (or) Extend the current partition to fill the unused disk space so that DataNode automatically increases the HDFS size?   ... View more

I don't think it is dynamically allocated, or at least it doesn't seem to be working. I've run out of space trying to load a ~70 GB file. How can I increase the capacity? ... View more

I'll look into it. I'll have to install gcc and then later Maven to run those shell scripts. Thanks for your input. ... View more