Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎08-29-2018

unable to stream data via flume and store them in hbase

[ Edited ]

Hello,

 

I'm new to Hadoop (Cloudera) and I'm just trying to stream tweets via flume and I want to save them in Hbase. However, I get exceptions and currently do not know what how to solve this problem.

 

I have a SplittingSerializer-0.0.1-SNAPSHOT.jar that I have stored in my Flume-ng/lib directory.

 

In my flume-env.sh file I have specified the following directory: FLUME_CLASSPATH="/opt/cloudera/parcels/lib/flume-ng/lib/SplittingSerializer-0.0.1-SNAPSHOT.jar"

 

In the twitter.conf file I have entered the following parameters:

TwitterAgent.sources  = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = hbaseSink

TwitterAgent.sources.Twitter.type     = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxx
TwitterAgent.sources.Twitter.consumerSecret = xxx
TwitterAgent.sources.Twitter.accessToken = xxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxx
TwitterAgent.sources.Twitter.keywords = IoT

TwitterAgent.sinks.hbaseSink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
TwitterAgent.sinks.hbaseSink.channel=MemChannel
TwitterAgent.sinks.hbaseSink.table=hbtweet
TwitterAgent.sinks.hbaseSink.columnFamily=tweet
TwitterAgent.sinks.hbaseSink.serializer=org.apache.flume.sink.hbase.SplittingSerializer
TwitterAgent.sinks.hbaseSink.serializer.columns=tweet:nothing

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

 

then I execute this command:

flume-ng agent -n TwitterAgent -c conf -f /opt/cloudera/parcels/lib/flume-ng/conf/twitter.conf

 

In HBase I have created a table:

create 'hbtweet', 'tweet'

 

I get the following error message:

18/09/10 22:21:56 INFO zookeeper.ClientCnxn: Opening socket connection to server mrhel/xx:xx. Will not attempt to authenticate using SASL (unknown error)

18/09/10 22:21:56 INFO async.HBaseClient: Need to find the -ROOT- region 18/09/10 22:21:56 INFO hbase.AsyncHBaseSink: waiting on callback

18/09/10 22:21:56 INFO zookeeper.ClientCnxn: Socket connection established to mrhel/xx:xx, initiating session 18/09/10 22:21:56 INFO zookeeper.ClientCnxn: Session establishment complete on server mrhel/xx:xx, sessionid = xx, negotiated timeout = 5000

18/09/10 22:21:56 ERROR zookeeper.ClientCnxn: Caught unexpected throwable java.lang.NoSuchMethodError: org.hbase.async.generated.ZooKeeperPB$MetaRegionServer.makeExtensionsImmutable()V at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer.(ZooKeeperPB.java:128) at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer.(ZooKeeperPB.java:62) at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer$1.parsePartialFrom(ZooKeeperPB.java:137) at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer$1.parsePartialFrom(ZooKeeperPB.java:132) at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer$Builder.mergeFrom(ZooKeeperPB.java:409) at org.hbase.async.generated.ZooKeeperPB$MetaRegionServer$Builder.mergeFrom(ZooKeeperPB.java:324) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:108) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162) at org.hbase.async.HBaseClient$ZKClient$ZKCallback.handleMetaZnode(HBaseClient.java:3753) at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3621) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)

 

Does anyone know how I can solve this problem?

Announcements