Support Questions
Find answers, ask questions, and share your expertise

yarn logs + blk_xxxxxx_xxxxxx does not exist or is not under Construction

We have spark cluster with the following details ( all machines are linux redhat machines )


2 name-node machines
2 resource-manager machines
8 data-node machines ( HDFS file-system)


We are running running spark streaming application

From the yarn logs we can see the following errors , example:

yarn logs -applicationId application_xxxxxxxx -log_files ALL

---2019-11-08T10:12:20.040 ERROR [][][] [org.apache.spark.scheduler.LiveListenerBus] Listener EventLoggingListener threw an exception
org.apache.hadoop.ipc.RemoteException( BP-484874736- does not exist or is not under Construction
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
at org.apache.hadoop.ipc.RPC$
at org.apache.hadoop.ipc.Server$Handler$
at org.apache.hadoop.ipc.Server$Handler$
at Method)
at org.apache.hadoop.ipc.Server$


we can see that - `8478399929292:-blk_1081495827_7755233` does not exist or is not under Construction


but what could be the reasons that yarn complain about this?


Expert Contributor

Hi Mike,


Can you do quick check below -


**BP-484874736- does not exist or is not under Construction


1. Are all Datanodes up and running fine within cluster

2. Check on the NN UI and see if any Datanode is NOT reporting blocks in Datanode tab or any Missing blocks reported on NN UI

3. You can run fsck [unless cluster is huge and loaded with data] and check of the block exist and which all nodes has the replica.


It might help to drill down the issue.



1.  all Datanodes are up and running fine 

2. I not see corrupted block or under replica 

3, We runs the fsck and hdfs is healthy 


any other possibility's? 




we also do the following


su hdfs

hadoop fsck / -files -blocks >/tmp/file


and we bot found the block - blk_1081495827_7755233 in the file - /tmp/file


so what is the reason that block removed?


Expert Contributor

1. Is the job failed due to above reason?  If "NO", then is the error occurring in logs eveything for other BP XXX also?

2. Can you check using fsck which nodes had copied of the BP specified above?

please send me the fsck cli that you want me to run 


Expert Contributor

If you know the file name then -


hdfs fsck /myfile.txt -files -blocks -locations



hdfs fsck |grep <blkxxx>




by the following


hdfs fsck / -files -blocks -locations | grep blk_xxxxxx_xxxxxx




su hdfs

hdfs fsck / -files -blocks -locations | grep blk_1081495827_7755233 


we not get any results 


so I guess its mean that blk_xxxxx_xxxx isnt exist in HDFS file-system 


what next ?


Expert Contributor

1. Is the job failed due to above reason?

If "NO" - then is it the error occurring displayed in logs for all spark jobs  or just for this job?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.