Support Questions

Find answers, ask questions, and share your expertise

I need help with send change event from hdfs to Kafka

avatar
New Contributor

Can someone help me with define steps for integrating Kafka + HDFS + HBase. I need a solution for send events to Kafka on any change in hbase and hdfs.

1 ACCEPTED SOLUTION

avatar
Super Guru

You would need to implement a custom Coprocessor (likely, a RegionObserver would be sufficient) to notify any external system of changes to HBase. This is tricky to do correctly, so I would recommend that you re-think your architecture and make sure this is the way you want to implement this.

https://hbase.apache.org/book.html#_types_of_coprocessors

(For example, you may find it easier to push all of your data through Kafka and instead send events to HBase and HDFS per your business rules)

View solution in original post

2 REPLIES 2

avatar
Super Guru

You would need to implement a custom Coprocessor (likely, a RegionObserver would be sufficient) to notify any external system of changes to HBase. This is tricky to do correctly, so I would recommend that you re-think your architecture and make sure this is the way you want to implement this.

https://hbase.apache.org/book.html#_types_of_coprocessors

(For example, you may find it easier to push all of your data through Kafka and instead send events to HBase and HDFS per your business rules)

avatar
New Contributor

Thanks, I going to push all data through Kafka.