I have a scenario where i need to consume data from Elastic search and do some processing (like filter, enriching etc). I have to store this processed information to Redis. I have implementd the flow like Scroll ES Processer -- > Execute script . This is working well.As a part of ETL operation, i have to record total pass & failed record count at end of the process
1. how can i track total count of success and failure
2. how can i implement the workflow to wait until all the data consumed from elastic search then update the success and failure count to DB?