Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is it a best practice to use nifi querydatabase table to export around several gb of data from sql server to s3 from 250 tables and use incremental update?

avatar
Expert Contributor

Just wondering can we achieve this requirement using Nifi processor. Is it designed for such historical data or bulk upload from 250 tables in sql server to s3.

Iam aware that querydatabasetable processor support incremental updates when a new record is inserted. How to configure this processor to sync with existing record updates and deletes i.e. do we give unique column so that nifi can identify when ever an existing record is updated then it needs to sync it to s3. How to achieve this with nifi. I beleive nifi is not designed for bulk upload.

I have another usecase for bulk upload to s3 from mongodb. Please let me know if nifi can be the right tool for these type of usecases.

If so i believe i need to write a shell script to copy 250 sql server tables to each file in s3 in json format? Plz confirm? whats the best practice to query 250 tables and export to s3?

Thank you.

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

@Matt Burgess will look into the data and get back to you Matt. Thank you very much:)

avatar
Expert Contributor

@Matt Burgess i think you missed this bit but just checking with you again is this an effective solution to import bulk batch data from SQL server or mongodb using nifi to Amazon S3 and also keeping it in sync with updates and deletes? Is nifi designed for this purpose? We are looking to run these sync updates or deletes as overnight jobs in nifi? Plz reply thank you