- last edited on
I am connecting to remote sftp server to get data in hdfs via distcp. I have been able to do that. However, the way distcp connects dosen't look secure.
My command is:
hadoop distcp -D fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem -D fs.sftp.user.<hostname>=username -D fs.sftp.password.<hostname>.<username>=-<password> -pp -update sftp://<hostname>:22/var/opt/bdp/xyz.csv hdfs://<node>/user/username/
Is there any more secure way to login in sftp server ? This is interactive login,here passwords can be read inline. Suggestions required for secure logon.
Not using public/private key setup method as it requires setup on all Datanodes and will cause issues whenever cluster scaling happens.
I know there might be way to pass a file with username and password. Please help me the params and format for the file