Reply
New Contributor
Posts: 4
Registered: ‎06-27-2017
Accepted Solution

Sqoop import with wallet

Hi all,

 

We are facing some problems related to the wallet utilization with Sqoop v1.4.6 in Cloudera CDH 5.8.5. Our scenario:

  • user folder: /home/myuser
  • wallet folder: /home/myuser/wallet (cwallet.sso, cwallet.sso.lck, ewallet.p12, ewallet.p12.lck, sqlnet.ora, tnsnames.ora)

The wallet is well configured as if I run

sqlplus /@MY_TNS_NAME

I can connect to the db without the password.

The problem is with sqoop. I can connect to the db and list all the tables inside using:

export HADOOP_OPTS="-Doracle.net.tns_admin=/home/myuser/wallet -Doracle.net.wallet_location=/home/myuser/wallet"sqoop list-tables --connect "jdbc:oracle:thin:@MY_TNS_NAME" 

If I try to dump a table into HDFS with the following command (as suggested here)

export HADOOP_OPTS="-Doracle.net.tns_admin=/home/myuser/wallet -Doracle.net.wallet_location=/home/myuser/wallet"sqoop import -D mapreduce.map.java.opts='-Doracle.net.tns_admin=. -Doracle.net.wallet_location=.' -files /home/myuser/wallet/cwallet.sso,/home/myuser/wallet/ewallet.p12,/home/myuser/wallet/sqlnet.ora,/home/myuser/wallet/tnsnames.ora  -libjars "/var/lib/sqoop/oraclepki.jar,/var/lib/sqoop/osdt_cert.jar,/var/lib/sqoop/osdt_core.jar,/var/lib/sqoop/sqljdbc4.jar" --m 1 --connect "jdbc:oracle:thin:@MY_TNS_NAME" --query "SELECT * FROM MYTABLE WHERE \$CONDITIONS" --target-dir '/tmp/mytabledump'

I get this error:

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: encountered a problem with the Secret Store. Check the wallet location for the presence of an open wallet (cwallet.sso) and ensure that this wallet contains the correct credentials using the mkstore utility: java.io.IOException: oracle.security.crypto.core.CipherException: Invalid padding string (or incorrect password)...

If I run the same command without using the wallet (providing manually user and password) it works.

 

Additional info:

  • ACLs allows the sqoop user to access /home/myuser
  • the /home/myuser/wallet folder owner group is the sqoop primary group (rwx permissions).

It seems there are some problems passing the wallet to the mappers. Any idea how to solve this?

 

New Contributor
Posts: 4
Registered: ‎06-27-2017

Re: Sqoop import with wallet

We managed to solve the issue.

 

The problem was related to the wallet.

 

As far as I know there are 2 types of wallet. The first type can be used only on the machine where it was created. The second type can be used on multiple machines.

 

Our problem is that our wallet was of the first type.

 

During map reduce the wallet is distributed among mappers and so with this "non sharable" wallet the mappers (located on different machines) are not able to use it.

 

We had to create the wallet with the command: 

 

mkstore -wrl "<wallet_folder>" -create

Hope our solution will help someone else in the future.

 

Stefano

Announcements