Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 12-18-2016 06:19 PM
Hello,
I created the following topology:
Config config = new Config();
config.setClasspath("/usr/hdp/2.5.0.0-1245/hbase/lib");
config.setDebug(true);
config.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);
Map<String, Object> HBConf = new HashMap<String,Object>();
HBConf.put("hbase.rootdir","hdfs://localhost:8020/apps/hbase/data");
HBConf.put("zookeeper.znode.parent", "/hbase");
config.put("HBCONFIG",HBConf);
//TEST HBASE
SimpleHBaseMapper mapper = new SimpleHBaseMapper()
.withRowKeyField("row")
.withColumnFields(new Fields("driverName"))
.withColumnFamily("events");
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word-spout", new WordGenerator());
builder.setBolt("pre-hive", new PrepareTuple()).shuffleGrouping("word-spout");
builder.setBolt("hbase-bolt", new HBaseBolt("driver_dangerous_event", mapper).withConfigKey("HBCONFIG")).shuffleGrouping("pre-hive");
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("HelloStorm", config, builder.createTopology());
This topology is built with three components:
WordGenerator - Just generates random word),
PrepareTuple - Prepare data to be inserted in HBase. It generates tree fields used in SimpleHBaseMapper class
public void execute(Tuple input) {
this.collector.emit(new Values(id,input.getString(0), "events" ));
id=id+1;
System.out.println("###PRE-HBASE### Emitting tuple:"+ Integer.toString(id) +","+ user.getNome());
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("row","driverName","events"));
}
The idea is to create a tuple to be written in the "driver_dangerous_event" HBASE table. Unfortunately, no rows are inserted in the table. This is the log of my topology but I can't see errors:
###PRE-HBASE### Emitting tuple:2,Word 1
7767 [Thread-16-pre-hive-executor[3 3]] INFO o.a.s.d.executor - Execute done TUPLE source: word-spout:4, stream: default, id: {}, [Word 1] TASK: 3 DELTA:
7776 [Thread-18-__acker-executor[1 1]] INFO o.a.s.d.executor - Preparing bolt __acker:(1)
7787 [Thread-20-__system-executor[-1 -1]] INFO o.a.s.d.executor - Preparing bolt __system:(-1)
7797 [Thread-18-__acker-executor[1 1]] INFO o.a.s.d.executor - Prepared bolt __acker:(1)
7800 [Thread-20-__system-executor[-1 -1]] INFO o.a.s.d.executor - Prepared bolt __system:(-1)
8341 [Thread-14-hbase-bolt-executor[2 2]] WARN o.a.h.u.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
8752 [Thread-14-hbase-bolt-executor[2 2]] INFO o.a.h.h.z.RecoverableZooKeeper - Process identifier=hconnection-0x2ea63a1 connecting to ZooKeeper ensemble=localhost:2181
9193 [Thread-14-hbase-bolt-executor[2 2]] INFO o.a.s.d.executor - Prepared bolt hbase-bolt:(2)
9193 [Thread-14-hbase-bolt-executor[2 2]] INFO o.a.s.d.executor - Processing received message FOR 2 TUPLE: source: pre-hive:3, stream: default, id: {}, [2, Word 1, events]
9198 [Thread-14-hbase-bolt-executor[2 2]] INFO o.a.s.d.executor - Execute done TUPLE source: pre-hive:3, stream: default, id: {}, [2, Word 1, events] TASK: 2 DELTA:
9198 [Thread-14-hbase-bolt-executor[2 2]] INFO o.a.s.d.executor - Processing received message FOR -2 TUPLE: source: __system:-1, stream: __tick, id: {}, [1]
12709 [Thread-22-word-spout-executor[4 4]] INFO o.a.s.d.task - Emitting: word-spout default [Word 1]
12710 [Thread-22-word-spout-executor[4 4]] INFO o.a.s.d.executor - TRANSFERING tuple [dest: 3 tuple: source: word-spout:4, stream: default, id: {}, [Word 1]]
Is There something wrong here?
Thanks
Created 12-18-2016 07:45 PM
FIXED. I updated the configuration as following:
Map<String, Object> HBConf = new HashMap<String,Object>();
HBConf.put("hbase.zookeeper.quorum", "server1, server2, server3");
/*Find server1,server2,server3 in your configuration ambari>HBASE>advanced>hbase.zookeeper.quorum */
HBConf.put("hbase.zookeeper.property.clientPort", "2181");
HBConf.put("hbase.master.port", "16000");
/*Find zookeeper.znode.parent in your configuration ambari>HBASE>advanced>zookeeper.znode.parent */
HBConf.put("zookeeper.znode.parent", "/hbase-unsecure");
config.put("HBCONFIG",HBConf);
Created 12-18-2016 06:37 PM
just to test could you please try after updating line
builder.setBolt("hbase-bolt", new HBaseBolt("driver_dangerous_event", mapper).withConfigKey("HBCONFIG")).shuffleGrouping("pre-hive");
with this line
builder.setBolt("hbase-bolt", new HBaseBolt("driver_dangerous_event", mapper).withConfigKey("HBCONFIG")).fieldsGrouping("pre-hive",new Fields("row"));
Created 12-18-2016 07:45 PM
FIXED. I updated the configuration as following:
Map<String, Object> HBConf = new HashMap<String,Object>();
HBConf.put("hbase.zookeeper.quorum", "server1, server2, server3");
/*Find server1,server2,server3 in your configuration ambari>HBASE>advanced>hbase.zookeeper.quorum */
HBConf.put("hbase.zookeeper.property.clientPort", "2181");
HBConf.put("hbase.master.port", "16000");
/*Find zookeeper.znode.parent in your configuration ambari>HBASE>advanced>zookeeper.znode.parent */
HBConf.put("zookeeper.znode.parent", "/hbase-unsecure");
config.put("HBCONFIG",HBConf);