Support Questions
Find answers, ask questions, and share your expertise

Sqoop Import with Secure Password

Sqoop Import with Secure Password

Contributor

Dear Team,

How to move data from a relational database system via sqoop command to Hadoop with encrypted password?

Regards,

NIlesh

6 REPLIES 6

Re: Sqoop Import with Secure Password

Mentor

Here you go

Here's a quick Sqoop command example:

sqoop import \
    -Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader \
    -Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop2 \
    --connect jdbc:mysql://example.com/sqoop \
    --username sqoop \
    --password-file file:///tmp/pass.enc \
    --table tbl

The important parameters to note are:

  1. org.apache.sqoop.credentials.loader.class - Credentials loader
  2. org.apache.sqoop.credentials.loader.crypto.alg – The Algorithm used to decrypt the file (default is AES/ECB/PKCS5Padding).
  3. org.apache.sqoop.credentials.loader.crypto.salt – The salt used to derive a key with the passphrase (default is SALT).
  4. org.apache.sqoop.credentials.loader.crypto.iterations – Number of PBKDF2 iterations (default is 10000).
  5. org.apache.sqoop.credentials.loader.crypto.salt.key.len – Derived key length (default is 128).
  6. org.apache.sqoop.credentials.loader.crypto.passphrase Passphrase used to derive key.

There are several other options available to you as well more.

Re: Sqoop Import with Secure Password

Contributor

Hi,

In above Sqoop command example,

1. What is pass.enc. How to create it?

2. ///tmp is from local disk or from Hadoop?

Re: Sqoop Import with Secure Password

Mentor

@Nilesh follow my link in my answer it has step by step

Re: Sqoop Import with Secure Password

Mentor

@Nilesh Sqoop whats your sqoop version ? 1.4.5

The most secure way of supplying password to the database is to save the password in a file on the users home directory with 400 permissions and specify the path to that file using the --password-file argument, and is the preferred method of entering credentials. Sqoop will then read the password from the file and pass it to the MapReduce cluster using secure means with out exposing the password in the job configuration. The file containing the password can either be on the Local FS or HDFS. For example:

$ sqoop import --connect jdbc:mysql://database.example.com/employees \ 
    --username harry 
--password-file ${user.home}/.password

Sqoop

Re: Sqoop Import with Secure Password

Mentor

@Nilesh I have a step by step with way to create password file. Link

Re: Sqoop Import with Secure Password

Expert Contributor

How do you create an encrypted password file for sqoop "pass.enc"? All you did was post a link on how to save it in clear text, which is not secure enough for production environments. Thanks