@Eyad Garelnabi According to the Hadoop Documentation, permissions checks for the superuser always succeed, even if you try to restrict them. The process (and group) used to start the namenode become the superuser and can always do everything within HDFS. ... View more

@ANSARI FAHEEM AHMED 1) If you hover your mouse over the "HDFS Disk Usage" widget (upper left hand corner) in the Ambari Dashboard it will show you the following details: DFS Used: Storage used for data Non-DFS Used: Storage used for things such as logs, shuffle writes, etc... Remaining: Remaining storage 2) From the command line you can also run "sudo -u hdfs hdfs dfsadmin -report", which will generate a full report of hdfs storage usage. 3) Finally, if you would like to check the disk usage for a particular folder (and sub folders), then you can use commands like "hadoop fsck", "hadoop fs -dus" or "hadoop fs -count -q". For an explanation of the differences between these commands as well as how to read the results please take a look at this post: http://www.michael-noll.com/blog/2011/10/20/understanding-hdfs-quotas-and-hadoop-fs-and-fsck-tools/ ... View more

@mkataria With HDFS Snapshots there is no actual data copying up front for a new snapshot. It is simply a pointer to a record in time (point-in-time). So when you first take a snapshot, your HDFS storage usage will stay the same. It is only when you modify the data that data is copied/written. This follows the Copy on Write (COW) concept. Please take a look at the below JIRA. IT contains the discussion that lead to the design and is quite informative. https://issues.apache.org/jira/browse/HDFS-2802 ... View more

@nejm hadj First I’ll answer your question and then I’ll make my recommendation. Answer: The name of the file does not matter. When setting up a Hive external table just specify the data source as the folder that will contain all the files (regardless of names). Details on setting up and external table: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/moving_data_from_hdfs_to_hive_external_table_method.html Details on reading/parsing JSON files into Hive: http://hortonworks.com/blog/howto-use-hive-to-sqlize-your-own-tweets-part-two-loading-hive-sql-queries/ (alternatively, you can convert JSON to CSV within NiFi. To do so, follow the NiFi portion of this example https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.html) Recommendation: HDFS prefers large files with many entries as opposed to many files with small entries. The main reason being that for each file landed on HDFS, file information is saved in the NameNode (in memory). If you’re putting each twitter message in a separate file you will quickly fill up your NameNodes’s memory and overload the server. I suggest you aggregate multiple messages into one file before writing to HDFS. This can be done with the MergeContent processor in Nifi. Take a look at the below screenshots showing how it would be set up. Also, take a look at the NiFi Twitter_Dashboard.xml example template (https://raw.githubusercontent.com/abajwa-hw/ambari-nifi-service/master/demofiles/Twitter_Dashboard.xml). You can import this into your NiFi by by clicking on Templates (third icon from right) which will launch the 'Nifi Flow templates' popup, and selecting the file. ... View more

You may want to try: https://hbasenet.codeplex.com/ ... View more