Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka-Storm-Hbase integration

Kafka-Storm-Hbase integration

New Contributor

Hi,

I need to integrate data coming from Kafka into Hbase using a Storm spout.

What are the ways of doing it?

Thanks.

5 REPLIES 5

Re: Kafka-Storm-Hbase integration

Expert Contributor

Re: Kafka-Storm-Hbase integration

@Rohan kapadekar See this http://hortonworks.com/hadoop/storm/#tutorials

This landing page has example for HBASE, Hive, Storm ,Kafka and other components.

Re: Kafka-Storm-Hbase integration

Mentor

@Rohan kapadekar please refer to the storm user guide as that has all the information you need to develop Kafka, hbase, HDFS and hive bolts and spouts.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/ch_using_storm.h...

Re: Kafka-Storm-Hbase integration

Guru
Even though there is integartion built into Storm, I often find it easier to write my own....

Do this on prepare()

Configuration config = HBaseConfiguration.create();

		config.set("hbase.zookeeper.quorum", Constants.zkHost);

		config.set("hbase.zookeeper.property.clientPort", Constants.zkPort);

		config.set("zookeeper.znode.parent", "/hbase-unsecure");

		try {

			HBaseAdmin hbaseAdmin = new HBaseAdmin(config);

			if (hbaseAdmin.tableExists("TransactionHistory")) {

			transactionHistoryTable = new HTable(config, "TransactionHistory");

			}else{

				HTableDescriptor tableDescriptor = new HTableDescriptor("TransactionHistory");

				HColumnDescriptor cfColumnFamily = new HColumnDescriptor("Transactions".getBytes());

		        tableDescriptor.addFamily(cfColumnFamily);

		        hbaseAdmin.createTable(tableDescriptor);

		        transactionHistoryTable = new HTable(config, "TransactionHistory");

			}

			hbaseAdmin.close();
		} catch (IOException e) {
			e.printStackTrace();
		}

Do this on execute()


Put transactionToPersist = new Put(Bytes.toBytes(transaction.getTransactionId()));
transactionToPersist.add(Bytes.toBytes("Transactions"), Bytes.toBytes("accountNumber"), Bytes.toBytes(transaction.getAccountNumber()));

Re: Kafka-Storm-Hbase integration

Guru

See answer below, it is often easier to write your own. I find the pre-built HBase bolts to be hard to use.