Hello @Eren    Mabe this kb can be useful https://community.cloudera.com/t5/Customer/HDFS-write-occasionally-fails-with-error-message-quot-Unable/ta-p/81420    The block chain replication for HDFS does not seem to be able to complete:    snapshot export:   2023-04-19 19:44:10,199 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_-9223372036854751104_9943 is COMMITTED but not COMPLETE(numNodes= 5 >= minimum = 3) in file /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508 2023-04-19 19:44:10,200 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call Call#31 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.0.146:52210 …. hadoop distcp: java.io.IOException: Unable to close file because the last block does not have enough number of replicas.     Then make sure that the destination cluster has enough number of datanodes available and with space based on the replication factor or the hbase file, in this case    /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508   You can reconfirm the RF of the file by running any of the following commands:    hdfs fsck  /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508   hdfs dfs -ls  /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508.    In the last one, the second column will tell the RF applied to the file.    You can also check in the destination cluster if the datanodes are distributed in racks, if so, check that each rack has enough number of datanodes available and with space.    Hope this helps.   ... View more

Hello @Vako   You may require further investigation from the AWS support team concerning this:    [ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error)   This error comes from S3 protocol, also this message from NN logs:   java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.   This means that the client reached the Namenode, so it wrote the file at metadata level in HDFS, but when the application tried to write the data to the different datanodes this fails for some reason, maybe you can check if Datanode's logs tell you something at the time of the issue, but I've always seen this issue caused by some problem/misconfiguration in the application side, also you can validate if there are no firewall rules blocking the datanode port for the server where you are running this AWS DataSync agent.   Hope this helps.    ... View more

Hello @hanumanth ,    If Zookeeper services look up and running, you may need to compare the Spark job failure timestamp against Zookeeper logs from the Leader sever. if there is not a visible issue from Zookeeper side you can check if the hbase client configurations were applied properly in the spark job configurations.   Also, confirm that the Hbase service is up and functional as well.    If the above does not help, you may want to raise a support ticket with the Spark component.  ... View more

What is the current HDP, CDH, or CDP version of this cluster? Is this issue present on every browser? Did it work as expected before?   ... View more