We have a requirement where in we have to read messages from Kafka topic using storm and insert them to database. But we can only do batch insertions... so to achieve this, we are collecting every message(tuple) into a collection and for every 3 seconds we run a thread/timer which persists this collection to a database as a batch update. but we would like to acknowledge only after a collection/batch of messages are persisted to the database, so that failovers can be handled and those failover messages can be re-consumed. Is there any workaround for this issue
Thanks a lot for your reply Artem Ervits. I have one more question. What happens a transaction fails while persisting to the database. we have code within a bolt to persist to database and if database is down for any reason, can we replay the tuple to persist the tuple/message to database
@anil k you can probably add some logic to retry the failed tuple, perhaps in another bolt, there are many possibilities. Why don't you open a new question and we can address it there. Also, if my answer was sufficient, please accept it to close out the thread.