Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

incremantal fetch from mongo db NIFI

avatar

i have  NIFI mongo db connection and need to fetch only new record , i have a long column (transactiondate) (yyyy-MM-dd HH:mm:ss) and need the new transactions in last 1 sec . 

8 REPLIES 8

avatar

hi @moahmedhassaan, it would really help if would provide more details about your flow, even how you query your data. Having and testing a MongoDB is not easy for everybody because there are not lots of people who have it available.
Nevertheless, there might a solution, not very efficient but it might do your thing:
Add a GenerateFlowFile Processor, where you configure a property like : ${now():format("yyyy-MM-dd HH:mm")}:${now():format("ss"):minus(1)}. Set this processor to run only on the primary node so you won't have to many generated files.

cotopaul_0-1678430557368.png

 


Send the success queue to your GetMongo Processor.

Within the GetMongo Processor, in the query property you write your query with the condition on transactiondate > The_property_defined_in_Generate_Flow_File.

cotopaul_1-1678430661056.png

 


Again this is not a very bright solution, but it could suite your needs until somebody with more experience can provide you with a solution 😊

avatar

thanks for your reply but query in getmongo processor not select statment it's json . 

i try query using {"TransactionHistory":"2022-12-21T13:24:34.945+00:00"} as the original  date format but with no result . 

avatar

maybe you can try with something like? I have no access to any MongoDBs right now, to test it myself 😞

{"TransactionHistory": {"$gt": "<your_value_here>"}} 

 

avatar

thanks  cotopaul for your comment , it seems fine for not date column like 

{"_id": {"$gt": 10}} 

 but for matching the timestamp not working 

avatar

this form work in mongo {"transactionDate": {"$gt": ISODate("2021-12-21")}}   but not in nifi 

avatar

Why does it not work in NiFi? What sort of an error message do you receive? NiFi basically executes what you set in the query property. If you can execute it in mongo, it should work from NiFi as well.

avatar

it give an error due to (ISODate)

avatar

can you maybe post the statement you are running in mongo + the result and afterwards the query you execute in NiFi and the entire error message you receive? Right now it seems to be an error regarding the format of the date/query. I am not that familiar with mongoDB but maybe in the meantime somebody with more experience will intervene.