Support Questions
Find answers, ask questions, and share your expertise

how to acknowledge tuples after doing batch insertions to a database

how to acknowledge tuples after doing batch insertions to a database

New Contributor

We have a requirement where in we have to read messages from Kafka topic using storm and insert them to database. But we can only do batch insertions... so to achieve this, we are collecting every message(tuple) into a collection and for every 3 seconds we run a thread/timer which persists this collection to a database as a batch update. but we would like to acknowledge only after a collection/batch of messages are persisted to the database, so that failovers can be handled and those failover messages can be re-consumed. Is there any workaround for this issue

4 REPLIES 4
Highlighted

Re: how to acknowledge tuples after doing batch insertions to a database

Mentor

Take a look at this example using elastic search. It receives a response from persist call and sends an ack Link. @anil k

Highlighted

Re: how to acknowledge tuples after doing batch insertions to a database

New Contributor

Thanks a lot for your reply Artem Ervits. I have one more question. What happens a transaction fails while persisting to the database. we have code within a bolt to persist to database and if database is down for any reason, can we replay the tuple to persist the tuple/message to database

Highlighted

Re: how to acknowledge tuples after doing batch insertions to a database

Mentor

@anil k you can probably add some logic to retry the failed tuple, perhaps in another bolt, there are many possibilities. Why don't you open a new question and we can address it there. Also, if my answer was sufficient, please accept it to close out the thread.

Highlighted

Re: how to acknowledge tuples after doing batch insertions to a database

Mentor

@anil k are you still having issues with this? Can you accept best answer or provide your workaround?