question incremantal fetch from mongo db NIFI in Support Questions

incremantal fetch from mongo db NIFI

moahmedhassaan — Thu, 09 Mar 2023 20:06:35 GMT

i have NIFI mongo db connection and need to fetch only new record , i have a long column (transactiondate) (yyyy-MM-dd HH:mm:ss) and need the new transactions in last 1 sec .

Re: incremantal fetch from mongo db NIFI

cotopaul — Fri, 10 Mar 2023 06:45:14 GMT

hi @moahmedhassaan, it would really help if would provide more details about your flow, even how you query your data. Having and testing a MongoDB is not easy for everybody because there are not lots of people who have it available.
Nevertheless, there might a solution, not very efficient but it might do your thing:
Add a GenerateFlowFile Processor, where you configure a property like : ${now():format("yyyy-MM-dd HH:mm")}:${now():format("ss"):minus(1)}. Set this processor to run only on the primary node so you won't have to many generated files.

Send the success queue to your GetMongo Processor.

Within the GetMongo Processor, in the query property you write your query with the condition on transactiondate > The_property_defined_in_Generate_Flow_File.

Again this is not a very bright solution, but it could suite your needs until somebody with more experience can provide you with a solution 😊

Re: incremantal fetch from mongo db NIFI

moahmedhassaan — Fri, 10 Mar 2023 09:39:29 GMT

thanks for your reply but query in getmongo processor not select statment it's json .

i try query using {"TransactionHistory":"2022-12-21T13:24:34.945+00:00"} as the original date format but with no result .

Re: incremantal fetch from mongo db NIFI

cotopaul — Fri, 10 Mar 2023 11:04:59 GMT

maybe you can try with something like? I have no access to any MongoDBs right now, to test it myself 😞

{"TransactionHistory": {"$gt": "<your_value_here>"}}

Re: incremantal fetch from mongo db NIFI

moahmedhassaan — Fri, 10 Mar 2023 12:06:42 GMT

thanks cotopaul for your comment , it seems fine for not date column like

{"_id": {"$gt": 10}}

but for matching the timestamp not working

Re: incremantal fetch from mongo db NIFI

moahmedhassaan — Fri, 10 Mar 2023 12:15:39 GMT

this form work in mongo {"transactionDate": {"$gt": ISODate("2021-12-21")}} but not in nifi

Re: incremantal fetch from mongo db NIFI

cotopaul — Fri, 10 Mar 2023 13:08:11 GMT

Why does it not work in NiFi? What sort of an error message do you receive? NiFi basically executes what you set in the query property. If you can execute it in mongo, it should work from NiFi as well.

Re: incremantal fetch from mongo db NIFI

moahmedhassaan — Sun, 12 Mar 2023 13:18:36 GMT

it give an error due to (ISODate)

Re: incremantal fetch from mongo db NIFI

cotopaul — Mon, 13 Mar 2023 09:38:02 GMT

can you maybe post the statement you are running in mongo + the result and afterwards the query you execute in NiFi and the entire error message you receive? Right now it seems to be an error regarding the format of the date/query. I am not that familiar with mongoDB but maybe in the meantime somebody with more experience will intervene.