question Re: PutSFTP - Need help in Support Questions

PutSFTP - Need help

hkh — Mon, 06 Feb 2023 13:55:03 GMT

I am having a simple flow where I am fetching the creds for SFTP from DB and then uploading a file using the putSFTP but I am getting UnknownHostException error on PutSFTP processor.

I see the creds are being fetched - I see them in logs and also tried connecting to the Server using FileZilla (creds are correct ..no space etc all good with that)..

Also I tried hardcoding the creds in PutSFTP and still it did not work.

can someone please guide me..

Re: PutSFTP - Need help

MattWho — Mon, 06 Feb 2023 15:27:25 GMT

@hkh
The NiFi service is owned by typically a service user. All components (processors, controller services, etc) are execute their code as that NiFi service user. So in the case of the putSFTP processor, that is the same.

So let's assume your NiFi process is owned by local system user "nifi". The putSFTP processor code is executed by the "nifi" user. So "nifi" is executing a ssh command to connect to a remote SFTP server as another user (your provided creds). Just as if you were executing this from command line on the host for the first time, the new SFTP server you are connecting to needs to be added to a known_hosts file for the "nifi" user. On command line, you would get a prompt to accept that new host. Within the putSFTP processor there is no way to accept that.

So from the NiFi server host as the NiFi service user, execute the ssh command to connect to the target SFTP server you are trying to interface with. When prompted to accept the add the new host to your known_hosts. Now go back to your dataflow in NiFi and try running the putSFTP processor again as see if you still encounter the same exception.

If this does not solve the issue, can you share your putSFYP processor configuration and the complete error message from the nifi-app.log?

You may also want to try putting the below putSFTP processor class in DEBUG in the NiFi logback.xml to see if any DEBUG output helps determine the issue.

org.apache.nifi.processors.standard.PutSFTP

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Re: PutSFTP - Need help

hkh — Mon, 06 Feb 2023 20:08:26 GMT

Please see below comments..

Re: PutSFTP - Need help

hkh — Mon, 06 Feb 2023 20:06:34 GMT

o.a.nifi.processors.standard.PutSFTP PutSFTP[id=e46aac15-0185-1000-0243-066468b818f4] Unable to transfer StandardFlowFileRecord[uuid=3a9e192c-7f34-48de-ace9-c144ac41e9d9,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1674478122732-215, container=default, section=215], offset=13832, length=62],offset=0,name=test.hl7,size=62] to remote host myhostname://test.net due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutSFTP[id=e46aac15-0185-1000-0243-066468b818f4]: java.io.IOException: Failed to obtain connection to remote host due to com.jcraft.jsch.JSchException: java.net.UnknownHostException: myhostname://test.net: java.io.IOException: Failed to obtain connection to remote host due to com.jcraft.jsch.JSchException: java.net.UnknownHostException: myhostname://test.net; routing to failure: java.io.IOException: Failed to obtain connection to remote host due to com.jcraft.jsch.JSchException: java.net.UnknownHostException: myhostname://test.net
java.io.IOException: Failed to obtain connection to remote host due to com.jcraft.jsch.JSchException: java.net.UnknownHostException: myhostname://test.net
at org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:515)
at org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:597)
at org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:130)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
at org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:122)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: myhostname://test.net
at com.jcraft.jsch.Util.createSocket(Util.java:394)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)

at org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:495)
... 17 common frames omitted
Caused by: java.net.UnknownHostException: myhostname://test.net
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util$1.run(Util.java:362)

Re: PutSFTP - Need help

hkh — Mon, 06 Feb 2023 20:08:00 GMT

Hi Matt,

Also have added the host to the known_host file as suggested in previous comment

have shared the details on error and configuration

Thanks.

Re: PutSFTP - Need help

hkh — Mon, 06 Feb 2023 20:09:55 GMT

For security reasons I have changes the hostname and also shared screenshot without disclosing any sensitive details.

Re: PutSFTP - Need help

hkh — Tue, 07 Feb 2023 19:38:17 GMT

Is there any alternative to putSFTP?

Re: PutSFTP - Need help

MattWho — Wed, 08 Feb 2023 18:02:40 GMT

@hkh I see that you have changed the hostname for security reasons, but I am confused by the "://" in your hostname. Is that present in the real hostname?

Thanks,

Matt

Re: PutSFTP - Need help

hkh — Wed, 08 Feb 2023 19:38:18 GMT

yes it is

Re: PutSFTP - Need help

MattWho — Thu, 09 Feb 2023 20:59:52 GMT

@hkh Based on RFC1123, the use of ": or /" in a hostname is not valid.
https://www.rfc-editor.org/rfc/rfc1123

From command line on the server where NiFi is running as the user who owns the NiFi running process, can you "ssh <username>@myhostname://test.net" successfully?

So I am still a bit confused on you having an SFTP-server running on a server with a hostname containing these characters. It may very well be that these invalid characters re resulting in the unknownHostException coming out of the Java SSH library used by the NiFi putSFTP processor.

As far as an alternative to putSFTP, that would require knowing more about your endpoint you are trying to write to in order to provide such suggestions.

Thank you,

Matt

Re: PutSFTP - Need help

hkh — Thu, 09 Feb 2023 22:33:14 GMT

Hi Matt,

Yes your suggestion worked 🙂

Thank you so much ..

Great catch ..issue was in the hostname as you shared in the resource..

Re: PutSFTP - Need help

hkh — Thu, 09 Feb 2023 22:34:09 GMT

Thank you Matt 🙂