question Re: Nifi Incremental ingest? in Archives of Support Questions (Read Only)

Nifi Incremental ingest?

frankmarit — Thu, 13 Oct 2016 01:19:26 GMT

I have some external rest apis that I have to query for data periodically using InvokeHTTP. I'd like to pass in the date as a query arg which I last extracted data to only retrieve the incremental changes. What are the best practices on how to do this with Nifi? Should I

* Use an external database table to update/query the last date?

* Is there a different built in mechanism I can use to accomplish this?

Currently, I'm just using ${now():toNumber():minus(86400000):format('yyyy-MM-dd')} to get the last day's date and passing this in to the rest api, but this isn't a good way to do it because if my daily load fails one day then the next day I will skip it.

Re: Nifi Incremental ingest?

TimothySpann — Thu, 13 Oct 2016 03:09:52 GMT

You could use the DistributedMapCache.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html

https://community.hortonworks.com/questions/35223/distributedmapcacheclientservice-nifi-wecrawlerxml.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService/index.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html

That's pretty easy since you are just using a date.

http://funnifi.blogspot.com/2016/04/inspecting-your-nifi.html

I also like storing that in HBase or an RDBMS or a small in-memory database like Redis, Ignite, Geode, but that's more work and another step.

Re: Nifi Incremental ingest?

frankmarit — Thu, 13 Oct 2016 05:08:50 GMT

Great! Thanks I will play with this. Is there a way to know when the whole workflow is complete? The last step in my workflow writes the data to a file, but it doesn't always come at once. Some items may be waiting in one of the queues or whatever. Suggestions?

Re: Nifi Incremental ingest?

TimothySpann — Thu, 13 Oct 2016 07:34:21 GMT

hit refresh look at data provenance

you can see numbers in queues if things are still processing

Re: Nifi Incremental ingest?

harsh1 — Wed, 26 Jul 2017 17:13:40 GMT

According to this https://community.hortonworks.com/questions/103459/clarifications-on-state-management-within-nifi-pro.html and my research -

I understand that DistributedMapCache is not actually distributed and it runs on individual nodes. If the node running the server fails then the data is gone. Also, it is a cache server so has an eviction strategy, though it gives the option of persistence directory but that does not solve anytime availability problem. When we want to store some temporary state then it may be good but for long term persistent state we should rather rely on Zookeeper for its distributed nature. Unfortunately, I could not find any processor for putting data in Zookeeper. Other option would be to use database or distributed storage like HDFS, S3 etc.

Please correct me if I am wrong anywhere.

PS: I have the same case where I want to get the data from an API and wants to store the time upto which I have already requested the data.

Re: Nifi Incremental ingest?

frankmarit — Thu, 27 Jul 2017 07:01:52 GMT

@Harsh ChoudharyAgreed. I came to the conclusion that the distributed map cache is too flakey to keep track of important things. We've seen it mysteriously fail several times and have since changed all our processes to use a database.

Re: Nifi Incremental ingest?

TimothySpann — Tue, 10 Oct 2017 01:15:17 GMT

There has been a major upgrade to cache in Apache NiFi 1.4 and now you can use Redis!