Solved
Datanode added but not seen by namenode
We recently added a two nodes to our cluster through ambari wizard, we installed datanode, nodemanager, Metrics Monitor, LogFeeder
The datanode/nodemanager are starting correctly by not live
topology_mappings.data was updated in both mnode and cnodes
cat /etc/hadoop/conf/topology_mappings.data
[network_topology]
cnode2.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.172=/default-rack
cnode5.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.169=/default-rack
cnode4.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.175=/default-rack
cnode3.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.67=/default-rack
cnode1.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.188=/default-rack
cnode6.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack
10.1.2.9=/default-rack
datanodes have 2 external disks to store hdfs data
[root@node6 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 200G 33M 200G 1% /grid/disk0
/dev/vdc 200G 33M 200G 1% /grid/disk
We are using hdp 2.6.5 with freeipa as ldap, we checked that everything was created successfully (principals, keytabs ...) but logs are showing some warnings/errors with kerberos
datanodes logs:
2022-06-06 10:45:39,357 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(227)) - Problem connecting to server: mnode0.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com/10.1.2.145:8020
2022-06-06 10:45:39,641 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(227)) - Problem connecting to server: mnode1.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com/10.1.2.106:8020
mnode logs:
2022-06-06 10:47:55,038 INFO ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client 10.1.2.169 threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User dn/cnode5.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com@2B87D4BC-6CF3-4350-AAF7-EFF7227D1AEF.DATALAKE.COM (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only accessible by dn/10.1.2.169@2B87D4BC-6CF3-4350-AAF7-EFF7227D1AEF.DATALAKE.COM]
