Member since
04-19-2020
2
Posts
0
Kudos Received
0
Solutions
09-04-2020
12:38 AM
Hi, I have a requirement where we have a template which uses SQL as source and SQL as destination and data would be more than 100GB for each table so here template will be instantiated multiple times based on tables to be migrated and also each table is partitioned into multiple flowfiles. How do we know when the process is completed? As here there will be multiple flowfiles we are unable to conclude as it hits end processor. I have tried using SitetoSiteStatusReportingTask to check queue count, but it provides count based on connection and its difficult to fetch connectionid for each connection then concatenate as we have large number of templates. Here we have another problem in reporting task as it provides data on all process groups which are available on NIFI canvas which will be huge data if all templates are running and may impact in performance even though I used avro schema to fetch only queue count and connection id. Can you please suggest some ideas and help me to achieve this? Thanks Sreeja
... View more
Labels:
- Labels:
-
Apache NiFi
04-19-2020
04:44 AM
Hi Manoj, We are using textfiles with separationchar as '|', but the problem we have embedded new lines in columns which is resulting data empty in hive as its considering as new line. Rest data is migrating perfectly fine. Could you please suggest us how to avoid new line characters in between column-data. Thanks&Regards Sreeja
... View more