Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to Rollback all Tables -if load fails in partial state

Highlighted

How to Rollback all Tables -if load fails in partial state

Explorer

XML->JSON->oracle tables. For example: I have 5 tables.

For some reason 3rd table load fails then 1,2,3 tables should be rollback by not sitting in partial state.

 

My flow Get the JSON file and form a required JSON format(JoltTransform) for 5 tables and route to all 5 tables(doing operations in each table - ADD/MODIFY/REMOVE based on JSON file content) 

for db insert - from routeAttributes ->convertJSONtoSQL ->putSQL

ADD- Insert,MODIFY-Update and REMOVE-Delete from table.

MPraveen_0-1585630540580.png

 

2020-03-31 14_51_19-NiFi Flow.png

2 REPLIES 2
Highlighted

Re: How to Rollback all Tables -if load fails in partial state

@MPraveen This adjustment to your Use Case & POC is going to get technical and complicated.   

 

When you have artifact events that operate individually, but the combination of all events are required for final steps, you have to implement some kind of check system.  You would need this check system to wait and ONLY execute the final steps, ONLY when all of the artifacts are confirmed to be ready.  For example: if table count = 5, execute final steps.    You could also use a similar method to rollback queries that are already completed, but I favor not executing any query until you are sure they all need to be executed.

 

To me this indicates some kind of NiFi Wait & Notify.   Your wait will be put into the final step, and notify will be sent as each artifact finishes.  When all finish, wait should allow the gate to open and all artifacts finish their final step.

 

Here is a community search for "wait notify".  

 

https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=fal...

 

Here is an example on GitHub:

 

https://gist.github.com/ijokarumawak/20125d663d2116c6dae1eecae8d7acbc

 

Another useful article:

 

https://medium.com/@rahulsoni4u/trigger-based-serial-data-processing-in-nifi-using-wait-and-notify-p...

 

 

I would recommend you spend some time outside of your flow, learning wait/notify, and then apply the knowledge to your POC.

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Re: How to Rollback all Tables -if load fails in partial state

Explorer

Great and Thanks for detailed answer. I will take a look and apply the logic. 

Don't have an account?
Coming from Hortonworks? Activate your account here