Sqoop throwing InvalidAlgorithmParameterException while initializing EncryptionManager

RubilksCube — Fri, 16 Sep 2022 11:43:09 GMT

We're using bouncyCastle for our encryption to a DB2 database. If we update java.security file in our default Oracle JVM (on our edgenode and all datanodes) so bouncyCastle is listed first everything works fine. However when our servers get updates (such as updates to Java) the java.security file gets replaced. As our servers have various levels of security on them it's like an act of congress to get the java.security file remodified.

To save time and avoid the hassle, we are attempting to use our own security file. I have this partially working with

-Djava.security.properties=/san-data/deploy/scanning.security

When I say partially working, I mean Sqoop reads the columns from the table and compiles a java class, it's when the mapreduce jobs kick off that I get

2017-06-07 13:24:25,148 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.lang.RuntimeException: com.ibm.db2.jcc.am.SqlException: [jcc][1071][10615][3.65.119] Caught java.security.InvalidAlgorithmParameterException while initializing EncryptionManager.  See attached Throwable for details. ERRORCODE=-4223, SQLSTATE=null
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:170)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: com.ibm.db2.jcc.am.SqlException: [jcc][1071][10615][3.65.119] Caught java.security.InvalidAlgorithmParameterException while initializing EncryptionManager.  See attached Throwable for details. ERRORCODE=-4223, SQLSTATE=null
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:223)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168)
        ... 10 more
Caused by: com.ibm.db2.jcc.am.SqlException: [jcc][1071][10615][3.65.119] Caught java.security.InvalidAlgorithmParameterException while initializing EncryptionManager.  See attached Throwable for details. ERRORCODE=-4223, SQLSTATE=null
        at com.ibm.db2.jcc.am.cd.a(cd.java:682)
        at com.ibm.db2.jcc.am.cd.a(cd.java:60)
        at com.ibm.db2.jcc.am.cd.a(cd.java:85)
        at com.ibm.db2.jcc.am.oc.a(oc.java:152)
        at com.ibm.db2.jcc.t4.b.u(b.java:2817)
        at com.ibm.db2.jcc.t4.b.b(b.java:821)
        at com.ibm.db2.jcc.t4.b.a(b.java:761)
        at com.ibm.db2.jcc.t4.b.a(b.java:422)
        at com.ibm.db2.jcc.t4.b.a(b.java:397)
        at com.ibm.db2.jcc.t4.b.<init>(b.java:335)
        at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:233)
        at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:199)
        at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:475)
        at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:116)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:215)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:216)
        ... 11 more
Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 2048 (inclusive)
        at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DHKeyPairGenerator.java:120)
        at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:675)
        at java.security.KeyPairGenerator.initialize(KeyPairGenerator.java:412)
        at com.ibm.db2.jcc.am.oc.a(oc.java:141)
        ... 25 more

How do I get the mapreduce jobs that Sqoop kicks off to use my security file?

I've tried using

-D mapreduce.map.java.opts="-Djava.security.properties=scanning.security" -D yarn.app.mapreduce.am.command-opts="-Djava.security.properties=scanning.security" -files /san-data/deploy/scanning.security

I've tried --verbose on Sqoop but that doesn't turn verbose on the mapreduce jobs so I don't know if bcprov-jdk15on-1.55.jar or my security file are copied out there and I don't know where "there" is so I don't known if the mapreduce.map.java.opts or yarn.app.mapreduce.am.command-opts are pointing to the correct location. Is there a way to get more information from the mapreduce jobs (I'm not seeing any helpfull information in yarn logs)?

Here's a better example of my command (sensitive info removed)

sqoop import -D mapreduce.map.java.opts="-Djava.security.properties=scanning.security" -D yarn.app.mapreduce.am.command-opts="-Djava.security.properties=scanning.security" -D mapreduce.map.maxattempts=2 -files /san-data/deploy/scanning.security --connect "jdbc:db2://servername:3740/db:queryDataSize=229375;securityMechanism=13;readOnly=true;" --username xxxx --columns "*" --table QUALIFIER.TBLPTS --where "1=1" --target-dir /data/subData/SYSTEM/raw --mysql-delimiters -m 1 --verbose --password

Re: Sqoop throwing InvalidAlgorithmParameterException while initializing EncryptionManager

RubilksCube — Thu, 08 Jun 2017 12:42:16 GMT

I figured out what my problem was, I mistakenly dropped

-libjars /var/lib/sqoop/bcprov-jdk15on-1.55.jar,/var/lib/sqoop/db2jcc-3.65.119.jar,/var/lib/sqoop/db2jcc_license_cisuz-3.65.119.jar,/var/lib/sqoop/db2jcc_license_cu-3.65.119.jar

from my command line string. Once I added that back in things worked.

question Re: Sqoop throwing InvalidAlgorithmParameterException while initializing EncryptionManager in Archives of Support Questions (Read Only)

Sqoop throwing InvalidAlgorithmParameterException while initializing EncryptionManager

Re: Sqoop throwing InvalidAlgorithmParameterException while initializing EncryptionManager