incremantal fetch from mongo db NIFI

[#1]

i have  NIFI mongo db connection and need to fetch only new record , i have a long column (transactiondate) (yyyy-MM-dd HH:mm:ss) and need the new transactions in last 1 sec . 

[#2]

hi @moahmedhassaan, it would really help if would provide more details about your flow, even how you query your data. Having and testing a MongoDB is not easy for everybody because there are not lots of people who have it available.
Nevertheless, there might a solution, not very efficient but it might do your thing:
Add a GenerateFlowFile Processor, where you configure a property like : ${now():format("yyyy-MM-dd HH:mm")}:${now():format("ss"):minus(1)}. Set this processor to run only on the primary node so you won't have to many generated files.

Send the success queue to your GetMongo Processor.

Within the GetMongo Processor, in the query property you write your query with the condition on transactiondate > The_property_defined_in_Generate_Flow_File.

Again this is not a very bright solution, but it could suite your needs until somebody with more experience can provide you with a solution 😊

[#3]

thanks for your reply but query in getmongo processor not select statment it's json . 

i try query using {"TransactionHistory":"2022-12-21T13:24:34.945+00:00"} as the original  date format but with no result . 

[#4]

maybe you can try with something like? I have no access to any MongoDBs right now, to test it myself 😞

{"TransactionHistory": {"$gt": "<your_value_here>"}} 

[#5]

thanks  cotopaul for your comment , it seems fine for not date column like 

{"_id": {"$gt": 10}} 

 but for matching the timestamp not working 

[#6]

this form work in mongo {"transactionDate": {"$gt": ISODate("2021-12-21")}}   but not in nifi 

[#7]

Why does it not work in NiFi? What sort of an error message do you receive? NiFi basically executes what you set in the query property. If you can execute it in mongo, it should work from NiFi as well.

[#8]

it give an error due to (ISODate)

[#9]

can you maybe post the statement you are running in mongo + the result and afterwards the query you execute in NiFi and the entire error message you receive? Right now it seems to be an error regarding the format of the date/query. I am not that familiar with mongoDB but maybe in the meantime somebody with more experience will intervene.