About sze

sze · ‎07-18-2018

It is perfectly fine to mix drive types on a node. Just tag the storage type ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for each directory specified in dfs.datanode.data.dir; see also https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml (search "dfs.datanode.data.dir")

sze · ‎07-18-2018

For each block, it only caches a single copy of the block. The number of block replicas to cache is the number of replicas from different blocks.

sze · ‎07-18-2018

All the file data will be visible if the writer has invoked hflush/hsync successfully; see also https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/Syncable.html

sze · ‎07-18-2018

> ... IllegalArgumentException: Wrong FS: hdfs://HDP05/data/IFRS9/prod, expected: hdfs://HDP39 ... This is a known bug already fixed by https://issues.apache.org/jira/browse/HDFS-11432. Sorry that you were hitting it. What is your hadoop/hdp version?

sze · ‎07-18-2018

@rama, thanks for reporting the issue. It is a bug since FileSystem client should show NullPointerException to users. Would you like to file a JIRA to Apache ( https://issues.apache.org/jira/projects/HDFS )? I am happy to help.

sze · ‎12-01-2017

What is the conf dfs.datanode.data.dir set to? I suspect it is set to different directories under the same physical disk.

sze · ‎11-30-2017

Is this problem reproducible? Or, just a one time thing?

sze · ‎11-28-2017

For recoverLease using CLI, see https://community.hortonworks.com/questions/146012/force-closing-a-hdfs-file-still-open-because-uncor.html?childToView=146021#answer-146021

sze · ‎11-28-2017

Thanks a lot! Ming and I have tried it. It works well.

sze · ‎11-27-2017

"Sleep and retry" is good way to handle the "not have enough number of replicas" problem. For the "already the current lease holder" problem, you may call DistributedFileSystem.recoverLease(Path) to force lease recovery. Hope it helps.

Online	Offline
Last Visited	‎07-18-2018 07:27 PM

Date Registered	‎07-05-2016 11:16 PM
Last Visited	‎07-18-2018 07:27 PM
Total Messages Posted	25
Total Kudos Received	45

Re: HDFS reports a capacity larger than the sum of...

Re: How to handle: Unable to close file because th...

Re: Does Performing a Manual Failover Ever Disrupt...

Re: Storage Policies: Multiple storage types on a...

Re: HDFS Centralized Cache and block replicas

Re: Suppose if a 500 Mb of file and I have stored...

Re: Snapshotdiff from a remote cluster is failing

Re: -rm: Fatal internal error java.lang.NullPoint...

Re: HDFS reports a capacity larger than the sum of...

Re: HDFS gest the wrong name for the datanode

Re: How to handle: Unable to close file because th...

Re: How do run dfs command as another user?

Re: How to handle: Unable to close file because th...