04-25-2017 11:28 AM
Components Used for data ingestion:
Kafka Channel -> Flume Custom Sink -> DB
1. Before restart of flume -> i need to export zookeeper offset for a kafka.source.groupId
2. Restart flume
3. import the offsets. -> reason: while restarting some messages fails at sink processing (custom sink) side, and I need to re-process them.
I am not using flume transactions rollback capability as for some errors(data issue) other that point 3, the message might be failed and rollbacked indefinately.
I wanted to know if there is a better approach available other the export/import offsets everytime I restart flume.
04-26-2017 08:25 AM