Reply
New Contributor
Posts: 11
Registered: ‎02-11-2014

End of stream Exception with HBASE and YSCB

I'm trying to execute YCSB tests in my cluster with CDH5 but every time I execute the tests I see this:

 

 

14/02/12 04:53:46 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x48ada913, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 04:53:47 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x48ada913 connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c0bcd, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 04:53:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x26f8bd0a, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 04:53:47 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x26f8bd0a connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c0bce, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 04:53:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x5212b36f, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 04:53:47 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5212b36f connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c0bcf, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 04:53:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x75fb3120, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 04:53:47 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x75fb3120 connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 04:53:47 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c0bd0, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException

 

 

And in zookeeper logs I also see this:

 

2014-02-12 04:54:40,386 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.138.9.43:47185 which had sessionid 0x144241f0b1c0c17
2014-02-12 04:54:40,386 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x144241f0b1c0c18, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:724)
2014-02-12 04:54:40,386 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.138.9.43:47186 which had sessionid 0x144241f0b1c0c18
2014-02-12 04:54:40,386 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x144241f0b1c0c15, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:724)
2014-02-12 04:54:40,387 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.138.9.43:47183 which had sessionid 0x144241f0b1c0c15
2014-02-12 04:54:40,387 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x144241f0b1c0c16, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:724)
2014-02-12 04:54:40,387 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.138.9.43:47184 which had sessionid 0x144241f0b1c0c16
2014-02-12 04:54:40,387 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x144241f0b1c0c1b, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:724)

 

 

I compiled ycsb with these options in the pom.xml

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.96.1.1-cdh5.0.0-beta-2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.2.0-mr1-cdh5.0.0-beta-2</version>
</dependency>

 

All the installation of hbase was managed by CDH, so I didn't do any manual installation.

 

Regards!

Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: End of stream Exception with HBASE and YSCB

At first, is the HBase working fine? For example, can you run hbase shell to create table, put some data, and so on?

Second, in your ycsb classpath, is there any different HBase jar file? Under your db/hbase/lib in the ycsb, do you have the same hbase and zookeeper jar files as those in your HBase?

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: End of stream Exception with HBASE and YSCB

Yes. I'm able to create a table that is needed by YCSB.

And when I run this command:

bin/ycsb load hbase -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s | tee -a workloada.dat

The table gets populated.

 

And the JAR's that I have under lib are:

hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar

zookeeper-3.4.5-cdh5.0.0-beta-2.jar

 

 

 

This is the output of the command:

YCSB Client 0.1
Command line: -dbLoading workload...
com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s -load
Starting test.
0 sec: 0 operations;
14/02/12 17:11:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:host.name=ip-10-138-9-43.ec2.internal
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_25
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-oracle-cloudera/jre
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/var/lib/hadoop-hdfs/ycsb-0.1.4/core/lib/core-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/slf4j-api-1.6.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/infinispan-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/voldemort-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/jdbc-binding/conf
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-54-virtual
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/lib/hadoop-hdfs/ycsb-0.1.4
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0xa74cb5e, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 17:11:48 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xa74cb5e connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c50af, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x321236f4, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 17:11:48 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x321236f4 connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c50b0, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x25b56115, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 17:11:48 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x25b56115 connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c50b1, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
14/02/12 17:11:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x51f6c99, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 17:11:48 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51f6c99 connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 17:11:48 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c50b2, negotiated timeout = 60000
Error accessing HBase table: java.io.IOException: java.lang.reflect.InvocationTargetException
0 sec: 0 operations; [UPDATE AverageLatency(us)=1] [INSERT AverageLatency(us)=929008.25] [CLEANUP AverageLatency(us)=19.75]
[OVERALL], RunTime(ms), 969.0
[OVERALL], Throughput(ops/sec), 0.0
[UPDATE], Operations, 4
[UPDATE], AverageLatency(us), 1.0
[UPDATE], MinLatency(us), 1
[UPDATE], MaxLatency(us), 1
[UPDATE], 95thPercentileLatency(ms), 0
[UPDATE], 99thPercentileLatency(ms), 0
[UPDATE], 0, 4
[UPDATE], 1, 0
[UPDATE], 2, 0
[UPDATE], 3, 0
[UPDATE], 4, 0
[UPDATE], 5, 0
[UPDATE], 6, 0

.......More Updates

[CLEANUP], 997, 0
[CLEANUP], 998, 0
[CLEANUP], 999, 0
[CLEANUP], >1000, 0
java -cp /var/lib/hadoop-hdfs/ycsb-0.1.4/core/lib/core-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/slf4j-api-1.6.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/infinispan-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/voldemort-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/jdbc-binding/conf com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s -load

Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: End of stream Exception with HBASE and YSCB

I just tried it myself and it didn't have such problem for me. It looks like a network related issue. Can you try to run ycsb from a box inside your HBase cluster?

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: End of stream Exception with HBASE and YSCB

I executed this from the box where zookeeper and hbase are installed. So basically is a 'localhost' communication.

 

I think the problem might be related with zookeeper, but I don't know how to debug it.

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: End of stream Exception with HBASE and YSCB

Ok. I deleted some properties that I added in order to avoid this problem I was saw the first time I tried to execute it:

 

<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
<description>The FileSystem for file: uris.</description>
</property>

<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>

 

 

So, basically this should be the original problem that I had:

 

YCSB Client 0.1
Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s -load
Loading workload...
Starting test.
0 sec: 0 operations;
14/02/12 19:17:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-2--1, built on 02/07/2014 18:28 GMT
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:host.name=ip-10-138-9-43.ec2.internal
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_25
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-oracle-cloudera/jre
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/var/lib/hadoop-hdfs/ycsb-0.1.4/core/lib/core-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/slf4j-api-1.6.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/infinispan-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/voldemort-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/jdbc-binding/conf
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-54-virtual
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-hdfs
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Client environment:user.dir=/var/lib/hadoop-hdfs/ycsb-0.1.4
14/02/12 19:17:26 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-138-9-43.ec2.internal:2181 sessionTimeout=60000 watcher=hconnection-0x2ffad8fe, quorum=ip-10-138-9-43.ec2.internal:2181, baseZNode=/hbase
14/02/12 19:17:26 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2ffad8fe connecting to ZooKeeper ensemble=ip-10-138-9-43.ec2.internal:2181
14/02/12 19:17:26 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-10-138-9-43.ec2.internal/10.138.9.43:2181. Will not attempt to authenticate using SASL (unknown error)
14/02/12 19:17:26 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-138-9-43.ec2.internal/10.138.9.43:2181, initiating session
14/02/12 19:17:26 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-138-9-43.ec2.internal/10.138.9.43:2181, sessionid = 0x144241f0b1c5c90, negotiated timeout = 60000
14/02/12 19:17:26 WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://ip-10-138-9-43.ec2.internal:8020/hbase/lib, ignored
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:197)
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:633)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:387)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:366)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:247)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:188)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:118)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:302)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
Exception in thread "Thread-3" java.lang.NoClassDefFoundError: org/codehaus/jackson/Versioned
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hbase.client.Operation.<clinit>(Operation.java:40)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:165)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:118)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:302)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.Versioned
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 27 more
Exception in thread "Thread-4" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.client.Scan
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:165)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:118)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:302)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
Exception in thread "Thread-2" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.client.Scan
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:165)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:118)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:302)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
Exception in thread "Thread-5" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.client.Scan
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:165)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:118)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:302)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
1 sec: 0 operations;
[OVERALL], RunTime(ms), 1482.0
[OVERALL], Throughput(ops/sec), 0.0
java -cp /var/lib/hadoop-hdfs/ycsb-0.1.4/core/lib/core-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/slf4j-api-1.6.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/infinispan-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/voldemort-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/jdbc-binding/conf com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s -load

Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: End of stream Exception with HBASE and YSCB

This exception shows you have some problem to init DynamicClassLoader. Do you have Hadoop/HDFS jars in your db/hbase/lib? You can copy all the jar files under your HBase lib over, and give it a try. Make sure the actual files are copied over for those symbol links.

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: End of stream Exception with HBASE and YSCB

Sorry, I don't understand wich directory is that one.

When you say db/hbase/lib, are you trying to say ycsb-0.1.4/hbase-binding/lib?

 

Cloudera Employee
Posts: 35
Registered: ‎09-04-2013

Re: End of stream Exception with HBASE and YSCB

Yes, that's right. I referred to my ycsb folder structure, which is a little different.

New Contributor
Posts: 11
Registered: ‎02-11-2014

Re: End of stream Exception with HBASE and YSCB

Same problem.

 

Here is the list of JAR's that is in the directory.

hadoop-annotations-2.2.0-cdh5.0.0-beta-2.jar
hadoop-auth-2.2.0-cdh5.0.0-beta-2.jar
hadoop-common-2.2.0-cdh5.0.0-beta-2.jar
hadoop-core-2.2.0-mr1-cdh5.0.0-beta-2.jar
hbase-binding-0.1.4.jar
hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar
hbase-common-0.96.1.1-cdh5.0.0-beta-2.jar
hbase-protocol-0.96.1.1-cdh5.0.0-beta-2.jar
slf4j-api-1.6.4.jar
zookeeper-3.4.5-cdh5.0.0-beta-2.jar

 

 

And you can see in this command, that these are actually being loaded.

java -cp /var/lib/hadoop-hdfs/ycsb-0.1.4/core/lib/core-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/zookeeper-3.4.5-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hadoop-core-2.2.0-mr1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-protocol-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hadoop-annotations-2.2.0-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-client-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hadoop-common-2.2.0-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-common-0.96.1.1-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/slf4j-api-1.6.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/hbase-binding/lib/hadoop-auth-2.2.0-cdh5.0.0-beta-2.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/infinispan-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/voldemort-binding/conf:/var/lib/hadoop-hdfs/ycsb-0.1.4/benchmark/site/media/protovis/lib/google-compiler/compiler-20100201.jar:/var/lib/hadoop-hdfs/ycsb-0.1.4/jdbc-binding/conf com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s -load

 

Announcements