About Tylenol

Tylenol · ‎06-15-2022

ah ! Can you try to run the below HDFS balancer command , The below command would move the blocks at a decent pace and would not affect the existing jobs nohup hdfs balancer -Ddfs.balancer.moverThreads=5000 -Ddfs.datanode.balance.max.concurrent.moves=20 -Ddfs.datanode.balance.bandwidthPerSec=10737418240 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.balancer.max-size-to-move=100737418240 -threshold 10 1>/home/hdfs/balancer/balancer-out_$(date +"%Y%m%d%H%M%S").log 2>/home/hdfs/balancer/balancer-err_$(date +"%Y%m%d%H%M%S").log you can also refer to the below doc if you need any tuning https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/data-storage/content/balancer_commands.html

DianaTorres · ‎06-06-2022

@clouderaskme Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!

PrernaU · ‎10-22-2021

@PabitraDas The objective is to copy data between two distinct clusters

vciampa · ‎06-03-2021

@Tylenol, Thank you very much for your help.

enirys · ‎05-03-2021

Hi, Port are configured correctly, i just restarted my vms and fixed the issue. But it happens many times, and now i'm seeing other errors in logs ERROR datanode.DataNode (DataXceiver.java:run(278)) - ovh-cnode19.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake.ovh:1019:DataXceiver error processing WRITE_BLOCK operation src: /10.1.2.106:34306 dst: /10.1.2.171:1019 java.io.IOException: Premature EOF from inputStream

Tylenol · ‎04-12-2021

Hi @JGUI , There is no requirement for deleting the data from the datanode that is going to be decommissioned. Once the DN is been decommissioned all the blocks in the DN would be replicated to a different DN. And is there any error that you are encountering while you are decommissioning ? Typically, HDFS would self-heal and would re-replicate the under-replicated blocks that are due to the DN that is been decommissioned. And NN would start replicating the blocks with the other two replication that is present in HDFS.

Online	Online
Last Visited	‎04-19-2024 12:59 AM

Member Since	‎05-24-2019 12:50 AM
Last Visited	‎04-19-2024 12:59 AM
Posts	55
Kudos received	1

Cloudera Community

Re: HDP HDFS balancer CLI not found

Re: ranger is not updating access audit logs

Re: HDP HDFS balancer CLI not found

Re: ranger is not updating access audit logs

Re: Configure viewFS with HDFS fedration on CDP-PC...

Re: Command Distcp is not working HDP 2.6.1 (kerbe...

Re: URGENT : Datanode started but not not live (de...

Re: Datanode recommissioning