Support Questions
Find answers, ask questions, and share your expertise

DISTCP via SFTP

DISTCP via SFTP

New Contributor

Hi All,

 

I am connecting to remote sftp server to get data in hdfs via distcp. I have been able to do that. However, the way distcp connects dosen't look secure.

My command is: 

hadoop distcp -D fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem -D fs.sftp.user.<hostname>=username -D fs.sftp.password.<hostname>.<username>=-<password> -pp -update sftp://<hostname>:22/var/opt/bdp/xyz.csv hdfs://<node>/user/username/

 

Is there any more secure way to login in sftp server ? This is interactive login,here passwords can be read inline. Suggestions required for secure logon. 

Not using public/private key setup method as it requires setup on all Datanodes and will cause issues whenever cluster scaling happens.

 

I know there might be way to pass a file with username and password. Please help me the params and format for the file