Without a Maximum Value Column, both GTF and QDT will continue to pull the same data. Without a column for which we could figure out the "last" value, how would they know to pull the rows "since the last time"? Often an increasing ID or a "last updated" timestamp column is used as the Maximum Value Column. In NiFi 1.7.0 (via NIFI-5143) and in the upcoming HDF 3.2 release, you can use the values in a column for GenerateTableFetch (versus the concept of "row number"), but without a Maximum Value Column you still can't do incremental fetch. The alternative would be a Change Data Capture solution (CDC), but these are vendor-specific and currently only CaptureChangeMySQL exists (although there is a proposed addition of a MS SQL Server version). ... View more

The solution provided by @Harald via link is good enough as a solution. The only prequesite is that the cluster should be on top of Azure Data Lake and not azure datastorage. If its a cluster then solution should be replicated on all nodes. If it's only Edge node then solution should only be applied only edge node. By solution I mean the solution in https://community.hortonworks.com/content/kbentry/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html. Thanks ... View more