Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Exception in createBlockOutputStream

avatar
New Contributor

I have installed HBase 0.94.15-cdh4.7.0 and having problem with creating a table from hbase shell. Hbase successfully creates two tables, but fails to create the third (hangs up). Here is my log from hbase:

 

2014-12-06 18:00:22,413 INFO org.apache.hadoop.hbase.master.handler.CreateTableHandler: Attempting to create the table UserGroups
2014-12-06 18:00:22,431 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1131)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1091)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)
2014-12-06 18:00:22,431 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning BP-1981556914-172.17.100.181-1406029837178:blk_5061750426017945617_24524723
2014-12-06 18:00:22,432 INFO org.apache.hadoop.hdfs.DFSClient: Excluding datanode 172.17.100.183:50010
2014-12-06 18:00:22,453 INFO org.apache.hadoop.hbase.regionserver.HRegion: creating HRegion UserGroups HTD == 'UserGroups', {NAME => 'Group', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://bman.dc2:8020/hbase/.tmp Table name == UserGroups
2014-12-06 18:00:22,472 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1305)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1131)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1091)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)
2014-12-06 18:00:22,473 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning BP-1981556914-172.17.100.181-1406029837178:blk_-6499242959049007209_24524726
2014-12-06 18:00:22,474 INFO org.apache.hadoop.hdfs.DFSClient: Excluding datanode 172.17.100.183:50010
2014-12-06 18:00:22,503 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
2014-12-06 18:00:22,504 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed UserGroups,,1417874422380.08f53c8488588eb88dd14e9613765937.
2014-12-06 18:00:22,516 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Added 1 regions in META
2014-12-06 18:00:22,516 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1 region(s) round-robin across 3 server(s)
2014-12-06 18:00:22,531 INFO org.apache.hadoop.hbase.master.AssignmentManager: bh1.dc2,60020,1417873861342 outstanding calls=1 of total=1
2014-12-06 18:00:23,518 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done

 

It seems that problem is caused not by the creation of table, but by some general network error. What may cause this problem?

2 REPLIES 2

avatar
> java.net.NoRouteToHostException: No route to host

This shows the network connection failed. You'd have to check the switch/router in between the hosts or if all hosts are connecting to each other with consistent host names and IP addresses.

Do you use DNS or /etc/hosts?
Regards,
Gautam Gopalakrishnan

avatar
New Contributor

Hello Gautam, thanks for reply.
The problem was in iptables at one of the data nodes.
It's a bit strange, because previous tables were created successfully, but turning iptables off solved the problem.