@lben  if you try to ssh to the target host from the NiFi host as the NiFi service user (User that owns the running NiFi), does it connect successfully or does it timeout at command line as well? If you made multiple connection attempts in a row from command line, do they all succeed? Is that the complete error from the nifi-app.log?  Was it followed with a stack trace that you can share? Thanks, Matt ... View more

@tcain  I see you are using a processor that is not part of the default Apache NIFi distribution (encodeProtobuf), so can't really comment on the performance of configurations specific to that processor. Can you share your configuration used in your 3 processors to include the Settings, Scheduling, Properties, and Relationship tabs?  That will help in understanding your current dataflow implementation setup.  I am very interested initially in the scheduling tab for each processor you are using.   2. The status bar jus above the the canvas in the NiFi UI will give a summary of all the component counts used on your canvas and their current status (enabled, disabled, running, stopped, invalid, etc).  Based on your description is expect on a "3" next to: 3. The Max Timer Driven Thread Count setting can be found by going to Global menu ---> Controller Settings. Although it does not sound like this may be issues since you have only 3 processors on your canvas and nothing else, correct? simply changing this value does not translate in a processor being able to do concurrent execution. 4. You can see your current JVM details (for standalone NiFi) via global menu --> summary --> system diagnostics (lower right) or (for Clustered NiFi), global menu --> cluster --> JVM tab. 5. ... 6. How large is the JSON you are sending through your pipeline? 7. What do you observe specific to lineage of your processed FlowFile.  You could run a provenance query.  Immediately after processing a FlowFile through your dataflow, you can run a data provenance query.  global menu ---> data provenance.   Screenshot of that return may be helpful as well to show execution times of each processor.  You can also click the small "view details" icon to the far left of each event for that FlowFile. From there you can see things like event duration and lineage duration.  This can help narrow down where specifically the slow down is occurring. I look forward to your detailed feedback and the additional information you can share here. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hkh I see that you have changed the hostname for security reasons, but I am confused by the "://" in your hostname.  Is that present in the real hostname?   Thanks, Matt ... View more

@alibugra007  First you have to ask yourself how you accomplish this task now manually? - Is there an SFTP server running on your Windows machine? - Does your Windows machine have network share you can mount on your Linux machine. The easiest approach may be to install NiFi on your windows machine and use the listSFTP and FetchSFTP NiFi processors to pull files from your Linux machine.  Then use the PutFile processor to write those files to a local directory on your windows machine where NiFi is running. Another option is to setup a NiFi on both the Linux machine and Windows machine and use NiFi's Remote Process Groups to pass FlowFiles from one NiFi to the other.  Then on Linux side use the listFile and Fetch File to pickup files and then on Windows side use PutFile to write those files to local Windows file system. Setting up NiFi on only the Linux machine requires you to have some way to connect to the Windows machine (SFTP server running on windows machine, Network share you can mount on your Linux machine, etc) If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@tcain  Here are some general things you should look at when encountering performance related issues: 1. How large are your dataflow(s) on yoru NiFi canvas? 2. How many running processors? 3. How large is yoru Max Timer Driven Thread count resource pool? (This is pool of threads sed by all processors to execute code, default is 10.  This should be incremented in small amounts as you monitor your CPU load average  to make sure you system is not CPU saturated.) 4. How is the health of your NiFi's JVM (Garbage collection happening very often, how long are the GC pauses)? 5. How is your disk I/O for the disk(s) hosting your NiFi content_repository, flowfile_repository, and provenance_repositories? 6. How many concurrent tasks are set on your processor furthest downstream with a backlog on its inbound connection (concurrent task should be carefully increased in strategic components.  Setting too high can have adverse affect on performance depriving other processors from being able to execute  optimally)? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hkh  The NiFi service is owned by typically a service user.  All components (processors, controller services, etc) are execute their code as that NiFi service user.  So in the case of the putSFTP processor, that is the same. So let's assume your NiFi process is owned by local system user "nifi".  The putSFTP processor code is executed by the "nifi" user.  So "nifi" is executing a ssh command to connect to a remote SFTP server as another user (your provided creds).  Just as if you were executing this from command line on the host for the first time, the new SFTP server you are connecting to needs to be added to a known_hosts file for the "nifi" user.   On command line, you would get a prompt to accept that new host.  Within the putSFTP processor there is no way to accept that.   So from the NiFi server host as the NiFi service user, execute the ssh command to connect to the target SFTP server you are trying to interface with.  When prompted to accept the add the new host to your known_hosts.  Now go back to your dataflow in NiFi and try running the putSFTP processor again as see if you still encounter the same exception.   If this does not solve the issue, can you share your putSFYP processor configuration and the complete error message from the nifi-app.log? You may also want to try putting the below putSFTP processor class in DEBUG in the NiFi logback.xml to see if any DEBUG output helps determine the issue. org.apache.nifi.processors.standard.PutSFTP   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@phaelax  This is very possible and very commonly done. NiFi's Remote Process Group (RPG) is commonly used (recommended method) to transfer NiFi FlowFiles between different NiFi instances or even different NiFi clusters: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Remote_Group_Transmission https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#remote_group_anatomy   When a direct connection between NiFi's can't be made, you can also use the mergeContent processor to "FlowFile Stream, v3" to merge numerous FlowFiles (includes FlowFile attributes) in to a single FlowFile that can be transferred by any means to another NiFi where it can be ingested and use the unpackContent processor to unplack that into the original FlowFiles with their original attributes.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hegdemahendra  When updating a NiFi variable a serious of steps needs to occur. Steps To Update Variables Identifying components affected Stopping affected Processors Disabling affected Controller Services Applying Updates Re-Enabling affected Controller Services Restarting affected Processors So in this process what can lead to this taking a long time to complete is the Stopping of processors and disabling of controller services using that updated variable.  When NiFi requests a component to stop it transitions to a "stopping" or "disabling" stage.  During this phase the component will not linger be scheduled and the process waits for any existing threads being executed by those components to complete. Those threads do not get interrupted.  So when this take aa long time or "infinite" amount of time, troubleshooting this would require getting a series of thread dumps to see which threads are long running or perhaps hung preventing the impacted components from competing the thread execution that blocks the component from transition to a fully stopped or disabled state.  Understand that nothing in  a thread dump is going to directly point back to a very specific processor.  So it is important in your troubleshooting that you know what processors use the variable you are updating and look for threads that appear in the complete series of multiple thread dumps that relate back to those component classes.  Also in cases were you see these long running calls, are they for a variables used consistently by the same set of components to help narrow your analysis. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt    ... View more