Member since
11-17-2016
63
Posts
7
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2140 | 11-23-2017 10:50 AM | |
4759 | 05-12-2017 02:13 PM | |
15455 | 01-11-2017 04:20 PM | |
8499 | 01-06-2017 04:03 PM | |
6286 | 01-06-2017 03:49 PM |
03-02-2021
06:09 PM
No worries @PR_224 Glad it's fixed : )
... View more
11-18-2020
12:34 AM
Did you get a solution to this issue? Would be grateful for your response.
... View more
12-06-2018
01:50 PM
To add, For sizing a datanode heap it's similar to namenode heap, its recommend 1GB per 1M blocks. As a block could be as small a 1byte or as large as 128MB, the requirement of heap space is the same.
... View more
12-14-2017
11:15 AM
Hi @Harsh J, Yesterday, I cleaned 50GB worth files from HDFS using fs -rm. The daily incoming size on HDFS is almost 13-15GB(including replication) however today again the size of dfs has increased almost 30-55 GB more. I dont understand why? Only, on one Datanode the dfs files generated almost 15GB. [root@DataNode1 finalized]# ls -lrt| grep "Dec 13" drwxr-xr-x 208 hdfs hdfs 4096 Dec 13 05:59 subdir130 drwxr-xr-x 195 hdfs hdfs 4096 Dec 13 06:52 subdir132 drwxr-xr-x 210 hdfs hdfs 4096 Dec 13 07:24 subdir134 drwxr-xr-x 188 hdfs hdfs 4096 Dec 13 07:32 subdir135 drwxr-xr-x 187 hdfs hdfs 4096 Dec 13 08:30 subdir138 drwxr-xr-x 210 hdfs hdfs 4096 Dec 13 09:09 subdir139 drwxr-xr-x 234 hdfs hdfs 12288 Dec 13 09:46 subdir173 drwxr-xr-x 234 hdfs hdfs 12288 Dec 13 10:07 subdir174 drwxr-xr-x 258 hdfs hdfs 12288 Dec 13 15:30 subdir211
... View more
11-23-2017
10:50 AM
I copied the older subdir* from dfs/dn/current/BP<...>/current/finalized to another storage mount/drive. This has not affected my data on hdfs. Please let me know if someone thinks there is a better way than this workaround. Thanks, Shilpa
... View more
08-15-2017
11:54 PM
1. hadoop fs -put <url folder> <target> 2. hadoop jar <deployment-jar> <classname> other_params
... View more
08-01-2017
09:53 AM
Ok @mbigelow thanks. I also reasearched more and found this hortonworks link, https://community.hortonworks.com/questions/45962/dataxceiver-error-processing-write-block-operation.html where they say we can ignore this error. This issue has been already fixed in 2.3 version of ambari but of course I am using CDH. Thanks, Shilpa
... View more
06-20-2017
09:58 AM
Hi @pdvorak, thanks a lot for you answer, I've already checked that page and it helped. Thanks again. DB
... View more
05-05-2017
10:08 AM
After increasing the Heap size of Datanode role on my NN, I have not seen Hprof files getting created. The issue is resolved. Thanks @mathieu.d
... View more
01-06-2017
03:49 PM
Hi @pdvorak, Thanks for your comments. The answer to all three questions are Yes. My iptables is turned off. I can ping my NN and tranverse hdfs. The problem was, the IP for hdfs sink I gave was private IP and once I changed it to public. It started streaming the data. So, the issue is resolved. 🙂
... View more