Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Storm-Hive Integration - could only be replicated to 0 nodes instead of minReplication (=1) Error

Highlighted

Storm-Hive Integration - could only be replicated to 0 nodes instead of minReplication (=1) Error

New Contributor

We have been attempting to load Data into Hive from Storm using the approach mentioned in http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-hive.html

This has been partially working and not working sometimes due to the below error.

2017-06-16 16:12:50.133 o.a.h.h.s.AbstractRecordWriter [ERROR] Unable to close org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater[hdfs://centos7264b-023.sweng.ncr.com:8020/apps/hive/warehouse/coreports.db/tlog_data/businessdate=2017-03-15/delta_2491803_2491807/bucket_00004] due to: File /apps/hive/warehouse/coreports.db/tlog_data/businessdate=2017-03-15/delta_2491803_2491807/bucket_00004 could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.

Storm logs and dfs admin Report are attached . Any help is appreciated . Its a blocking Issue to us.

report.png

workerlog.zip

1 REPLY 1

Re: Storm-Hive Integration - could only be replicated to 0 nodes instead of minReplication (=1) Error

New Contributor

namenode-datanode-logs.zip

Adding to above post We were looking the nameNode & dataNode logs, both of logs contain exception block. we could analyze that datanode is not able to write block to one of a datanode, the exception is "DiskOutOfSpaceException" and below is log stack trace about the exception

"2017-06-17 02:42:30,752 INFO  datanode.DataNode (DataXceiver.java:writeBlock(839)) - opWriteBlock BP-580032694-153.71.16.23-1491895068551:blk_1074111403_370899 received exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of space: The volume with the most available space (=90115424 B) is less than the block size (=134217728 B).
2017-06-17 02:42:30,752 ERROR datanode.DataNode (DataXceiver.java:run(278)) - 153.71.16.23:50010:DataXceiver error processing WRITE_BLOCK operation  src: /153.71.16.23:36283 dst: /153.71.16.23:50010
org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of space: The volume with the most available space (=90115424 B) is less than the block size (=134217728 B).
at org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:95)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:67)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.chooseVolume(FsVolumeList.java:80)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextVolume(FsVolumeList.java:109)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1332)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:207)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:674)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:745)"

When we check disk space in Hadoop cluster, noticed that DFS Remaining% :15.94% (5.22GB) for node 153.71.16.23. though we are persisting small chunk of data to hive (<200MB).

Can anyone help understand why datanode is unable to write to available space?

Don't have an account?
Coming from Hortonworks? Activate your account here