why don't you write 20 insert into query in append mode in the same sql file ';' separated, In this case you don't have to use union. And probably won't encounter heap space issue.   ... View more

Hello,   I'm not sure to understand your problem, but i'll try to answer to "Files ingested into HDFS but I don't see Files in HDFS".   When you process a flowfile with a processor, the processor will route the flowfile to relationship according with what it managed to do with the flowfile.   The PutHDFS processor have two relationship : success and failure. If the processor managed to put the flowfile to HDFS, the flowfile is routed to the relationship success and you can continue to process it. This unless you check automatically terminate the relationship success, then NiFi won't do anything more with the flowfile. Same for the failure relationship.   Here you automatically terminate both success and failure relationship. For success relationship this is ok, because your file was put to HDFS. But for failure relationship, the file is not put to HDFS and you don't handle the failure.   Just route the failure relationship to any other processor, you don't even need to use it at this point, you just want to see if flowfile are route to the failure relationship, and so if the queue corresponding to this relationship is filled. You can also see the failure if a red error box appear in the top right corner of the processor, but previous method if to understand the concept of relationship routes. If so, the flowfile are not put to HDFS, probably because of a bad configuration of the PutHDFS processor.    In this case, i'll be happy to help you with this configuration (properties tab).       ... View more

@Sru111 , If possible, please update your nifi version to 1.11.4 or above. You can find load balancing option there. Otherwise, stick to your plan of using primary node only for FetchSFTP processor. You can still do it in your nifi using Remote Process Groups. But, it will become really complex with that. ... View more

Hi @Sru111 , For FetchSFTP processor, change the Bulletin level to NONE. Then, that error won't appear. If there are any other errors like authentication error or communication timeout, bulletin won't appear for them as well. But, the flowFiles will still be routed to the respective relations. ... View more

Thank you for posting @Sru111 . The community has an automated spam filter which sometimes can errantly flag a valid post for various reasons. We have removed this flag and your post is now showing on the community as expected. We also removed the duplicates for you.    If this happens again, you can use the "Report inappropriate content" feature to alert moderators to the issue.    Click on the options menu (signified by three vertical dots) next to the post. Click on the "Report Inappropriate Content" option On the next page enter a comment describing the issue and click "Notify Moderator. Submitting the report will notify the community moderation team so the first available moderator can review the post and take the appropriate action.    ... View more