About jing

jing · ‎12-02-2016

Mainly two benefits: 1. Avoid unnecessary copy for renamed files/directories. If we renamed a large directory on the source side, "distcp -update" cannot detect the rename thus will copy the whole renamed directory as a new one. 2. More efficient copy list generation. "distcp -update" needs to scan the whole directory and detect identical files during the copy process. Thus the copy list generation may take a long time for a big directory. Using snapshot diff based approach can greatly decrease this workload in case of an incremental sync scenario. However, snapshot based distcp requires very careful snapshot management on both the source and target clusters. E.g., the target cluster must not have any modification between two copies. Otherwise the diff may not be applied correctly.

jing · ‎06-03-2016

First you may need to figure out the root cause for the read slowness: network issue? Slow disk? You can identify the corresponding DataNode that serves the data and then check its metrics to help debugging the issue. In the meanwhile, if the read is "position read", i.e., the read is called through API read(long, byte[], int, int), you can enable hedge read in DFSClient by setting the configuration "dfs.client.hedged.read.threadpool.size" to a non-zero number. Hedge read allows the reader to start reading from another DataNode (since there are usually 3 replicas) before the first read attempt finishes, if the reader thinks the first DataNode it read from is slow.

jing · ‎06-02-2016

Usually when you see "Cannot obtain block length for LocatedBlock", this means the file is still in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes. There are multiple possibilities here, .e.g., there may be temporary network connection issue between the reader and the DataNodes, or the original writing failed some while ago and the under-construction replicas are somehow missing. In general you run fsck command to get more information about the file. You can also trigger lease recovery for further debugging. Run command: hdfs debug recoverLease -path <path-of-the-file> -retries <retry times> This command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state.

jing · ‎05-28-2016

First you can check the hdfs audit log which tells the client name that created the file. Then based on the client name you may go and search the yarn application log to identify which job was writing the file.

jing · ‎05-28-2016

If the automatic failover is enabled in the cluster, to trigger the failover from Ambari, do not restart the active NameNode but shutdown the zkfc daemon on the active NN machine (note that to restart the NN in a large cluster may take a long time). Restart that zkfc daemon after the failover happens.

jing · ‎05-28-2016

Good question, @Mingliang Liu! There are many reasons for NN slowness: 1. GC. The GC may be caused by large workload like block reports, large amount of RPC calls, wrong heap settings, etc. The GC can be identified from NN gc log. The large amount block reports can be identified by checking NN log and the JMX metrics, the RPC load can be verified through JMX metrics and also hdfs audit log. 2. Slow editlog persistence. This may be caused by NameNode local disk issue, or sometimes network issue between NameNode and JournalNodes, or JournalNode disk issue. This usually can be identified from NN/JN log. 3. Slow editlog reading. This usually happens in the standby NN. From the NN log you can easily compute the reading speed of the editlog tailer inside of the standby NN. 4. Known/Unknown HDFS bugs. Usually you can take multiple thread dumps of the NN, and check if some thread is blocked by certain operation. If you find anything suspicious, ask for help here or send email to Apache hdfs/hadoop user mailing list.

jing · ‎05-28-2016

+1 for Joy's answer. This is just a health check from Ambari agent and it is harmless (although annoying maybe 🙂 ).

jing · ‎05-28-2016

Answers by @Sagar Shimpi and @Lester Martin look pretty good to me. Some further explanations: How does snapshots help for Disaster Recovery? What are the best practices around using snapshots for DR purposes? Especially trying to understand when data is directly stored on HDFS, Hive data and HBase data If you're using the current distcp for DR (i.e., using distcp copying data from one cluster to your backup cluster), you have an option to utilize snapshot to do incremental backup so as to improve the distcp performance/efficiency. More specifically, you can choose to take snapshots in both the source and the backup cluster and use -diff option for the distcp command. Then instead of blindly copying all the data, the distcp will first compute the difference between the given snapshots, and only copy the difference to the backup cluster. As I understand, no data is copied for snapshots, but only metadata is maintained for the blocks added/ modified / deleted. If that’s the case, just wondering what happens when the comamnd hdfs dfs -rm /data/snapshot-dir/file1 is run. Will the file be moved to the trash? If so, will the snapshot maintain the reference to the entry in trash? Will trach eviction has any impact in this case? Yes, if you have not skipped the trash, the file will be moved to the trash, and in the meanwhile, you can still access the file using the corresponding snapshot path. How does snapshots work along with HDFS quotas. For example, assume a directory with a quota of 1GB with snapshotting enabled. Assume the directory is closer to its full quota and a user deleted a large file to store some other dataset. Will the new data be allowed to be saved to the directory or will the operation be stopped because the quota limits have been exceeded? No, if the file belongs to the snapshot (i.e., the file was created before a snapshot was taken), you will not release quota by deleting it. You may have to delete some old snapshots or increase your quota limit. Also in some old hadoop versions you may find the snapshots also affect the namespace quota usage in a strange way, i.e., sometimes deleting a file can increase the quota usage. This has been fixed by the latest version of HDP.

Online	Offline
Last Visited	‎03-18-2017 12:15 AM

Member Since	‎10-26-2015 05:28 PM
Last Visited	‎03-18-2017 12:15 AM
Posts	10
Kudos received	29

Cloudera Community

Re: distcp update vs distcp update with snapshots

Re: How to make data read faster in HDFS

Re: Cannot obtain block length for LocatedBlock

Re: Possible reasons that cause slow name node (NN...

Re: distcp update vs distcp update with snapshots

Re: How to make data read faster in HDFS

Re: Cannot obtain block length for LocatedBlock

Re: Check opening files on HDFS

Re: Name node Manual failover

Re: Possible reasons that cause slow name node (NN...

Re: DataXceiver error processing unknown operation...

Re: Snapshots, Backup and DR