Support Questions
Find answers, ask questions, and share your expertise

flume with hbase with avro

flume with hbase with avro

Explorer

We have data coming into hdfs through flume. This data is serially encoded. We use the avro schema to decode it.

Now we want to put this data from flume into a hbase table instead of hdfs.

We use below parameter to do serial decoding based on avro schema :

ragent.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json

Will this same argument work when the target is hbase instead of hdfs? along with hbase table, hbase table column family etc?

Actually for some reason the data is not going in properly into hbase, and I am not sure if this avro schema is the issue.

Appreciate the insights.

1 REPLY 1
Highlighted

Re: flume with hbase with avro

Explorer

this is the flume configuration hbase sink part :

r_hbase.sinks.sink1.channel = channel1

r_hbase.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink

r_hbase.sinks.sink1.table = mev:rtable

r_hbase.sinks.sink1.columnFamily = me_data

r_hbase.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer r_hbase.sinks.sink1.serializer.schemaURL = hdfs://host.com:8020/tmp/avroschemas-new/rtable.json r_hbase.sinks.sink1.serializer.columns = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10

Don't have an account?