Member since
07-05-2016
25
Posts
45
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1070 | 12-01-2017 01:25 AM | |
10629 | 11-27-2017 10:51 PM | |
676 | 03-13-2017 08:45 PM |
07-18-2018
07:27 PM
2 Kudos
It is perfectly fine to mix drive types on a node. Just tag the storage type ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for each directory specified in dfs.datanode.data.dir; see also https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml (search "dfs.datanode.data.dir")
... View more
07-18-2018
07:23 PM
2 Kudos
For each block, it only caches a single copy of the block. The number of block replicas to cache is the number of replicas from different blocks.
... View more
07-18-2018
07:07 PM
All the file data will be visible if the writer has invoked hflush/hsync successfully; see also https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/Syncable.html
... View more
07-18-2018
07:01 PM
> ... IllegalArgumentException: Wrong FS: hdfs://HDP05/data/IFRS9/prod, expected: hdfs://HDP39 ... This is a known bug already fixed by https://issues.apache.org/jira/browse/HDFS-11432. Sorry that you were hitting it. What is your hadoop/hdp version?
... View more
07-18-2018
06:43 PM
@rama, thanks for reporting the issue. It is a bug since FileSystem client should show NullPointerException to users. Would you like to file a JIRA to Apache ( https://issues.apache.org/jira/projects/HDFS )? I am happy to help.
... View more
12-01-2017
01:25 AM
2 Kudos
What is the conf dfs.datanode.data.dir set to? I suspect it is set to different directories under the same physical disk.
... View more
11-28-2017
12:25 AM
1 Kudo
For recoverLease using CLI, see https://community.hortonworks.com/questions/146012/force-closing-a-hdfs-file-still-open-because-uncor.html?childToView=146021#answer-146021
... View more
11-27-2017
10:51 PM
3 Kudos
"Sleep and retry" is good way to handle the "not have enough number of replicas" problem. For the "already the current lease holder" problem, you may call DistributedFileSystem.recoverLease(Path) to force lease recovery. Hope it helps.
... View more