Member since
07-30-2020
219
Posts
45
Kudos Received
60
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
429 | 11-20-2024 11:11 PM | |
486 | 09-26-2024 05:30 AM | |
1081 | 10-26-2023 08:08 AM | |
1852 | 09-13-2023 06:56 AM | |
2126 | 08-25-2023 06:04 AM |
10-04-2023
11:30 PM
@Noel_0317 The directory /hadoop/dfs/name/ might be your Namenode data directory that contains the metadata in the form of fsimage and edits. So won't recommend deleting it if that's the case. You can confirm if this directory is indeed the NN data directory by checking the HDFS configuration. If the cluster is working and still taking writes, you can verify if the Namenode Data dir has been changed to a different mount point if the latest data available on it is from July.
... View more
09-29-2023
07:25 AM
@Noel_0317 If you want to know how the Datanode got upto 705GB, you will need to do a du at the Linux filesystem level for the datanode blockpool. For ex : du -s -h /data/dfs/dn/current/BP-331341740-172.25.35.200-1680172307700/ /data/dfs/dn/ ==> Datanode data dir BP ==> Blockpool used by the datanode The above should return 705GB. The Blockpool will contain the subdir which would be holding the File blocks present on this specific datanode. When you run 'hdfs dfs -du' it takes the entire HDFS storage into account.
... View more
09-13-2023
06:56 AM
@newtocm you can't pause the Balancer. You can kill it and start it again and it will try to balance the rest of the DFS data remaining to be balanced.
... View more
08-25-2023
06:04 AM
Generally, the size of data blocks would be 128mb across all the Datanodes. But, if you have small files then you might see smaller blocks on some Datanodes as well. So Datanodes with different disk spaces would have uneven "Number of Blocks" and the Balancing happens based on the difference in the DFS usage and not by the difference in block count.
... View more
08-24-2023
04:57 AM
Did you make any changes at the KDC end prior to seeing this issue? Are there any other services hosted on this node that are working fine?
... View more
08-24-2023
03:33 AM
@Crash you can set up load balance at a disk level for the Datanodes. Refer : https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/admin_dn_storage_balancing.html
... View more
08-17-2023
12:51 AM
Hi @Ben1996, As this is very specific use case, I don't see any Cloudera doc which could cover this. The basic requirements would be to import the hdfs module and provide the HDFS configs. For ex, https://tahiriamine9.medium.com/python-hdfs-cd822199799e#:~:text=Let's%20have%20an%20example%20of,into%20HDFS%20with%20CSV%20format.
... View more
08-07-2023
05:42 AM
I would suggest keeping one jar file at a time and checking. 1) /home/hadoop/hbase/hbase-2.5.5/lib/hbase-shaded-netty-4.1.4.jar!/META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so 2) /home/hadoop/hbase/hbase-2.5.5/lib/phoenix-client-hbase-2.5-5.1.3.jar!/META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
... View more
08-04-2023
12:38 AM
1 Kudo
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/phoenix_installation.html#concept_ysq_t4n_c3b
... View more
08-04-2023
12:36 AM
@Noel_0317 Do you have rack awareness configured for the Datanodes? Also, check for any disk-level issues on the datanode. Try enabling Debug for block placement : log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy=DEBUG
log4j.logger.org.apache.hadoop.hdfs.protocol.BlockStoragePolicy=DEBUG
... View more