I'm afraid that the GenerateTableFetch does not help me either because it uses the highest value on a column to determine whether to read new added records in the SQL server. In my case, I'm trying to extract the already existing records that have recently been modified. If there is no processor than can directly deal with this, i may add a column to the table view which displays the time when the record was last modified (i.e. LastModificationDateTime) and use this column to retain the "Maximum-value Columns" value. Back to the second part of my doubt, regarding the updating of this record in HDFS, how can I deal with this? I have the following approach in mind: 1 - Fetch and Read the parquet file where the "ChangedID" was stored. 2 - Modify the record in that file. 3 - Save up or replace that file in HDFS Is this approach correct? Is there any better solution?
... View more
