About alcbarczak

alcbarczak · ‎12-05-2017

Thanks for the answer Szetszwo, indeed that is the case. However, I have done that in the past with another cluster (different versions of Ambari) seemed to work fine. The reason I did that was that the nodes had 4 physical disks, and I mount them in the same directories that HDFS data goes (using a single config for HDFS for all datanodes). The server only has a single large disk, so I should have probably used a separate configuration for it. I'll try that and report back, thank again for your help. Andre.

alcbarczak · ‎12-05-2017

Thanks for your reply Venkat. I checked 1) and 2), and can confirm that all points to augustus.massey.ac.nz and IP 192.168.1.108, and even stopping and starting Ambari does not change the report from HDFS. I then tried to update the hostname as suggested in 4), and the answer (from the logs) is: 05 Dec 2017 14:31:04,108 ERROR [main] HostUpdateHelper:561 - Exception occurred during host names update, failed org.apache.ambari.server.AmbariException: Hostname(s): it431066.massey.ac.nz was(were) not found. at org.apache.ambari.server.update.HostUpdateHelper.validateHostChanges(HostUpdateHelper.java:197) at org.apache.ambari.server.update.HostUpdateHelper.main(HostUpdateHelper.java:544) That is what I expected, as I could not find any reference to it431066 machine anywhere in the configuration files nor in any of the Ambari repository. For example, "select host_name from hosts": it427066.massey.ac.nz augustus.massey.ac.nz Again, no mention of it431066. It is a very strange problem, there must be something in one of the installation scripts that is catching an IP address from the network rather than using the one in /etc/hosts. The wrong hostname/IP is in sequence to the master server's IP number, that's the only sense I could make of the issue. I might just clean up everything and start the installation again in an isolated network and see how it goes. Thanks for your help. Andre.

alcbarczak · ‎12-03-2017

Hi Szetszwo, thanks for the reply. It is reproducible. Also it prevents me to add any other datanode to this cluster because any new node will attempt to use the same name. I would like to know where to change the HDFS name manually rather than let the system decide. Finally, I only discovered about this issue because I saw the HDFS report (from command line) and the HDFS name on the logs, it does not show on Ambari dashboard. Thanks for any clues.

alcbarczak · ‎11-28-2017

When creating a new Ambari datanode, the name showing in the logs and in the HDFS report differs from the hostname of the node. The hdfs dfsadmin -report command: Live datanodes (2): Name: 131.100.200.83:50010 (IT431066.massey.ac.nz) Hostname: augustus.massey.ac.nz Decommission Status : Normal Configured Capacity: 2442941495296 (2.22 TB) DFS Used: 131072 (128 KB) Non DFS Used: 0 (0 B) DFS Remaining: 2044510578688 (1.86 TB) DFS Used%: 0.00% DFS Remaining%: 83.69% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 28 16:42:20 NZDT 2017 Last Block Report: Tue Nov 28 16:35:35 NZDT 2017 Name: 131.100.200.82:50010 (IT427066.massey.ac.nz) Hostname: IT427066.massey.ac.nz Decommission Status : Normal Configured Capacity: 37791439071232 (34.37 TB) DFS Used: 1163448320 (1.08 GB) Non DFS Used: 0 (0 B) DFS Remaining: 35613520156672 (32.39 TB) DFS Used%: 0.00% DFS Remaining%: 94.24% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 28 16:42:20 NZDT 2017 Last Block Report: Tue Nov 28 15:16:47 NZDT 2017 The Name in the first line differs from the actual Hostname. The IP address is also wrong, the node has a completely different IP address (192.168.1.108). There is no reference to the IP or to the hostname of the computer that shows up in the HDFS Name in the first line. Any clues or help would be appreciated.

alcbarczak · ‎11-28-2017

I am installing a new cluster for research and teaching. After installing and configuring, got the following report (hdfs dfsadmin -report): Configured Capacity: 37791439071232 (34.37 TB) Present Capacity: 35614683604992 (32.39 TB) DFS Remaining: 35613520156672 (32.39 TB) DFS Used: 1163448320 (1.08 GB) DFS Used%: 0.00% Under replicated blocks: 36 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (1): Name: xxx.xxx.xxx.xx:50010 (xxx.massey.ac.nz) Hostname: xxx.massey.ac.nz Decommission Status : Normal Configured Capacity: 37791439071232 (34.37 TB) DFS Used: 1163448320 (1.08 GB) Non DFS Used: 0 (0 B) DFS Remaining: 35613520156672 (32.39 TB) DFS Used%: 0.00% DFS Remaining%: 94.24% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 28 15:34:44 NZDT 2017 Last Block Report: Tue Nov 28 15:16:47 NZDT 2017 However, the machine only has about 12 TB of physical disk (df -k): /dev/sda2 990298620 790814264 149157060 85% / /dev/sda1 523248 3700 519548 1% /boot/efi /dev/sdb1 10544486348 2109284 10010942624 1% /hadoop As one can see , we only have around 11TB free space, but the HDFS reports 32.39TB. I already clean up (really cleaned up, used the phyton clean up script, removed every deb package, deleted every user related to ambari/HDFS, deleted every configuration file and logs etc) and reinstalled ambari from scratch three times, and the result is the same. Stranger yet, there is another machine with the same hardware where we installed the previous version of Ambari and that works correctly. Ambari version: 2.6.0 HDFS version: 2-6-3-0-235 (hadoop etc) Thanks for any help or clues that may help. Andre.

Online	Offline
Last Visited	‎12-05-2017 02:01 AM

Member Since	‎11-28-2017 02:28 AM
Last Visited	‎12-05-2017 02:01 AM
Posts	5

Cloudera Community

Cloudera Community

Re: HDFS reports a capacity larger than the sum of...

Re: HDFS gest the wrong name for the datanode

Re: HDFS gest the wrong name for the datanode

HDFS gest the wrong name for the datanode

HDFS reports a capacity larger than the sum of the...