- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
I need help with send change event from hdfs to Kafka
- Labels:
-
Apache Hadoop
-
Apache HBase
-
Apache Kafka
Created ‎08-03-2017 04:15 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can someone help me with define steps for integrating Kafka + HDFS + HBase. I need a solution for send events to Kafka on any change in hbase and hdfs.
Created ‎08-03-2017 04:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You would need to implement a custom Coprocessor (likely, a RegionObserver would be sufficient) to notify any external system of changes to HBase. This is tricky to do correctly, so I would recommend that you re-think your architecture and make sure this is the way you want to implement this.
https://hbase.apache.org/book.html#_types_of_coprocessors
(For example, you may find it easier to push all of your data through Kafka and instead send events to HBase and HDFS per your business rules)
Created ‎08-03-2017 04:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You would need to implement a custom Coprocessor (likely, a RegionObserver would be sufficient) to notify any external system of changes to HBase. This is tricky to do correctly, so I would recommend that you re-think your architecture and make sure this is the way you want to implement this.
https://hbase.apache.org/book.html#_types_of_coprocessors
(For example, you may find it easier to push all of your data through Kafka and instead send events to HBase and HDFS per your business rules)
Created ‎08-04-2017 06:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks, I going to push all data through Kafka.
