Skip to main content
moahmedhassaan
moahmedhassaan
Silver ⭐️⭐️⭐️
March 9, 2023
Question

incremantal fetch from mongo db NIFI

i have  NIFI mongo db connection and need to fetch only new record , i have a long column (transactiondate) (yyyy-MM-dd HH:mm:ss) and need the new transactions in last 1 sec . 

8 replies

cotopaul
cotopaul
Platinum ⭐️
March 10, 2023

hi @moahmedhassaan, it would really help if would provide more details about your flow, even how you query your data. Having and testing a MongoDB is not easy for everybody because there are not lots of people who have it available.
Nevertheless, there might a solution, not very efficient but it might do your thing:
Add a GenerateFlowFile Processor, where you configure a property like : ${now():format("yyyy-MM-dd HH:mm")}:${now():format("ss"):minus(1)}. Set this processor to run only on the primary node so you won't have to many generated files.

cotopaul_0-1678430557368.png

 


Send the success queue to your GetMongo Processor.

Within the GetMongo Processor, in the query property you write your query with the condition on transactiondate > The_property_defined_in_Generate_Flow_File.

cotopaul_1-1678430661056.png

 


Again this is not a very bright solution, but it could suite your needs until somebody with more experience can provide you with a solution 😊

Translate
moahmedhassaan
moahmedhassaanAuthor
Silver ⭐️⭐️⭐️
March 10, 2023

thanks for your reply but query in getmongo processor not select statment it's json . 

i try query using {"TransactionHistory":"2022-12-21T13:24:34.945+00:00"} as the original  date format but with no result . 

Translate
cotopaul
cotopaul
Platinum ⭐️
March 10, 2023

maybe you can try with something like? I have no access to any MongoDBs right now, to test it myself 😞

{"TransactionHistory": {"$gt": "<your_value_here>"}} 

 

Translate
moahmedhassaan
moahmedhassaanAuthor
Silver ⭐️⭐️⭐️
March 10, 2023

thanks  @cotopaul for your comment , it seems fine for not date column like 

{"_id": {"$gt": 10}} 

 but for matching the timestamp not working 

Translate
moahmedhassaan
moahmedhassaanAuthor
Silver ⭐️⭐️⭐️
March 10, 2023

this form work in mongo {"transactionDate": {"$gt": ISODate("2021-12-21")}}   but not in nifi 

Translate
cotopaul
cotopaul
Platinum ⭐️
March 10, 2023

Why does it not work in NiFi? What sort of an error message do you receive? NiFi basically executes what you set in the query property. If you can execute it in mongo, it should work from NiFi as well.

Translate
moahmedhassaan
moahmedhassaanAuthor
Silver ⭐️⭐️⭐️
March 12, 2023

it give an error due to (ISODate)

Translate
Terms & ConditionsAccessibility statement