@Prabhu Muthaiyan Glad that it helped you. Happy Hadooping!! ... View more

@Jonathan Samelson Non DFS space will be flushed out in if more memory is needed for processing few jobs. Also there will be an interval set in the configuration for clean up as well. Depending on that the memory will be cleared and used by hadoop. In your case when processing 50MB, it doesn't cost much memory than compared to processing 2.2 GB. Based on the size of file used for processing memory will be allocated/de-allocated. I guess it might answer your question. Below link might give some insights on allocation of memory. https://books.google.co.in/books?id=H3mvcxPeUfwC&pg=PA114&lpg=PA114&dq=dfs.datanode.du.reserved+example&source=bl&ots=pYyIud-Ix9&sig=qAkLTAkAtWCdITL1DiNqwEXrBqU&hl=en&sa=X&ved=0ahUKEwjgkvXZnpLTAhVHqI8KHVSRAgYQ6AEIPTAF#v=onepage&q=dfs.datanode.du.reserved%20example&f=false ... View more

@Jonathan Samelson Exactly. Non DFS is a reserved space allocated for hadoop services. Which means it will not used for data storage but however this space will be used by hadoop like an intermediate layer for various process/jobs which are triggered in hadoop. Similar to staging layer in typical ETL processing. ... View more

Hi @Jonathan Samelson Please refer this link https://community.hortonworks.com/questions/92839/hdfs-dropping-hive-table-is-not-freeing-up-memory.html#answer-92872 ... View more

@Saikrishna Tarapareddy Could you help me with your query? Is it throwing an error when you are using it in hive or the data which you were expecting is not replicated by the code? ... View more

@Christopher Daumer Create an external hive table with ORC and point it to your ORC file location. CREATE EXTERNAL TABLE IF NOT EXISTS mytable (col1 bigint,col2 bigint) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS ORC location '<ORC File location'; I dont think you can convert an ORC into csv directly. But you can write an UDF to do it. ... View more

Hi @Maeve Ryan It seems Disk Usage (Non DFS Used) is higher than DFS Used. DFS Used is your data in data node where as Non DFS Used is not exactly the data but some kind of log files which will be held by Yarn or other hadoop services. Check for huge files available in the disk. Then remove those files which will free free up your memory .Verify whats the size of DFS used before and after deleting the data file from HDFS. That will give an amount of memory available after the deletion. Memory calculation for non DFS: Non DFS Used = Configured Capacity - Remaining Space - DFS Used du -hsx * | sort -rh | head -10 helps to get the top 10 huge files. ... View more

Thanks @mqureshi ... View more