@Jay Kumar SenSharma Thanks jay for your response , in the lecture where I am experiencing the issue fs.permissions.umask-mode is set 022 but in other cluster where we are not getting the issue value if the same is 077 . ... View more

@Anurag Mishra can you check your kerberos ticket? # klist -e ... View more

@Anurag Mishra Something like this: # curl -s http://hdfcluster3.example.com:8088/ws/v1/cluster/scheduler | grep 'queueName' | python -c "import sys, json; print json.load(sys.stdin)['scheduler']['schedulerInfo']['queues']['queue'][0]['queueName']" # curl -s http://hdfcluster3.example.com:8088/ws/v1/cluster/scheduler | grep 'queueName' | python -c "import sys, json; print json.load(sys.stdin)['scheduler']['schedulerInfo']['queues']['queue'][1]['queueName']" # curl -s http://hdfcluster3.example.com:8088/ws/v1/cluster/scheduler | grep 'queueName' | python -c "import sys, json; print json.load(sys.stdin)['scheduler']['schedulerInfo']['queues']['queue'][2]['queueName']" Using a simple Shell script we can keep increasing the index in the same Curl command like ['queue'][3], ['queue'][4] . ... View more

@anurag The query failed because of the HDFS delegation token was not found in the cache and it has been cancelled by the running job. And the other job which has already started using that Delegation token, found that HDFS delegation token is already cancelled so the query failed with same error message. To resovlve this you need to add the property "tez.cancel.delegation.tokens.on.completion=false" Add in Custom tez-site via Ambari ... View more

@Sandeep Nemuri Hi Sandeep , Thanks for your response , but could you please explain what is the root cause of this issue , since from the falcon user also I tried to read hdfs and I was getting the same issue as I am doing this from hive user . however in falcon reading hdfs through webhdfs had solved the issue . I was creating a falcon cluster entity from falcon cli . I wanted to understand the root cause because I am experiencing the same issue from different users as well. Thanks in Advance. ... View more

@Anurag Mishra Please take a moment to login and click the "accept" link on the answer if you think the above helped address your question. ... View more

@Anurag Mishra When you use the following commands to push the data into your Hive table in new cluster hdfs dfs -mv /part2 /part2_old hdfs dfs -mv /part1 /part2 Your Hive engine and metastore don't get any intimation that a new partition has been added. It's simply a data copy/move operation on HDFS and Hive don't have any idea about it. A show partitions operation on your table in new cluster won't show anything. show partitions <your table name>; //Should not return anything You can tell your Hive engine to look into the HDFS and identify if some new data has been added outside of Hive by using the following command. msck repair table <your table name> Now if you do a show partitions, you shall be able to see the partitions that you just "created" by using the HDFS commands. Also, a select operation should work fine. If this reply helps you understand and fix your issue, please mark it as Accepted so that other community users can benefit from it. ... View more

Here is the other way Backing Up and Restoring HDFS Metadata Backing Up HDFS Metadata Using Cloudera Manager HDFS metadata backups can be used to restore a NameNode when both NameNode roles have failed. In addition, Cloudera recommends backing up HDFS metadata before a major upgrade. Minimum Required Role: (also provided by Full Administrator) This backup method requires you to shut down the cluster. Note the active NameNode. Stop the cluster. It is particularly important that the NameNode role process is not running so that you can make a consistent backup. Go to the HDFS service. Click the Configuration tab. In the Search field, search for "NameNode Data Directories" and note the value. On the active NameNode host, back up the directory listed in the NameNode Data Directories property. If more than one is listed, make a backup of one directory, since each directory is a complete copy. For example, if the NameNode data directory is /data/dfs/nn, do the following as root: # cd /data/dfs/nn # tar -cvf /root/nn_backup_data.tar . You should see output like this: ./ ./current/ ./current/fsimage ./current/fstime ./current/VERSION ./current/edits ./image/ ./image/fsimage If there is a file with the extension lock in the NameNode data directory, the NameNode most likely is still running. Repeat the steps, starting by shutting down the NameNode role. Restoring HDFS Metadata From a Backup The following process assumes a scenario where both NameNode hosts have failed and you must restore from a backup. Remove the NameNode, JournalNode, and Failover Controller roles from the HDFS service. Add the host on which the NameNode role will run. Create the NameNode data directory, ensuring that the permissions, ownership, and group are set correctly. Copy the backed up files to the NameNode data directory. Add the NameNode role to the host. Add the Secondary NameNode role to another host. Enable high availability. If not all roles are started after the wizard completes, restart the HDFS service. Upon startup, the NameNode reads the fsimage file and loads it into memory. If the JournalNodes are up and running and there are edit files present, any edits newer than the fsimage are applied. ... View more