Member since
07-03-2024
4
Posts
1
Kudos Received
0
Solutions
07-23-2024
06:41 AM
I referred to the other article because of the custom listSFTP processor. This would make it possible to use only one PG with the workflow "move file from A to B", read the job configuration by any other processor, put the config values on a flowfile and pass it to this listSFTP processor.
... View more
07-23-2024
06:38 AM
Copying a PG 500 times to just apply different parameters contexts, sounds not like an ideal solution. Also having all those workflow-identical PGs in a single canvas would make it hard to stay on top on things. But as far as I understand that is the only solution in NiFi I guess.
... View more
07-23-2024
05:43 AM
It is not only different paths. We also have multiple hosts/credentials. Each customer has usually different environments on separate hosts where we connect via SFTP to transfer some files (csv/Excel). These files come from multiples source SFTP accounts (same server, but differents creds per customer) and also sometimes from third parties (other servers). So what we need is also to set the Host name and credentials dynamically, based on a specific job configuration like: SourceHost: SourceHost_A SourceCreds: SourceCreds_A SourcePath: SourcePath_A SourceFilePattern: SourceFilePattern_A DestHost: DestHost_A DestCreds: DestCreds_A DestPath: DestPath_A These are the minimum configuration parameters which must be set dynamically. This describes a file transfer from A to B. We have hundreds of such transfers running involving ~300 different sftp accounts. Most of them are basic without any need for changes to the data. Sometimes changing file names and of course more complex transformation is needed. But in the majority of cases it is just basic moving the files from A to B.
... View more
07-03-2024
01:54 AM
1 Kudo
Hi guys, I am currently evaluating if NiFi can replace one or two of our tools (Talend OS/VisualCron). Therefor I need to find out how NiFi can be used to transfer (mainly "move") files from one sftp source to another. We have at the moment around 500 jobs in Visual Cron doing this. There are 40-50 servers in use and the jobs differ by file pattern and path. The closest solution I have found so far would be a custom processor mentioned in this post. But it seems outdated and need to get some fixes before it can be successfully build again. Either I could store all server and job configurations in an external file (yaml/xml) to create flowfiles or as preferred to store this data directly in NiFi - but not sure how. The benefit I see against VisualCron to let NiFi do the job is his queue system. This will make our transfers a lot more stable which is not at the moment. Do you think NiFi would be the right choice for this and can help me to find a suitable solution without creating 500 flows in one canvas? I want to have only one since all are almost the same beside some parametrization.
... View more
Labels:
- Labels:
-
Apache NiFi