Oozie workflow with Sqoop action and --password-alias option does not work via Hue

[alvinhenrick]

Hi,

      I am having this issue where encrypted password stored inside JKS is not working when invoking sqoop action from oozie workflow via Hue. Please find the exception.

    Product Version : Sqoop version: 1.4.6-cdh5.5.2 

991 [uber-SubtaskRunner] DEBUG org.apache.sqoop.tool.BaseSqoopTool  - Enabled debug logging.
2992 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Reflecting credential provider classes and methods
2992 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found org.apache.hadoop.security.alias.CredentialProvider
2993 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found org.apache.hadoop.security.alias.CredentialProviderFactory
2993 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found CredentialProvider#createCredentialEntry
2993 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found CredentialProvider#flush
2994 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found Configuration#getPassword
2994 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Found CredentialProviderFactory#getProviders
2994 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.password.CredentialProviderHelper  - Resolving alias with credential provider path set to jceks://hdfs/user/test_user/my.pwd.jceks
3036 [uber-SubtaskRunner] DEBUG org.apache.sqoop.Sqoop  - Error resolving password  from the credential providers 
java.lang.RuntimeException: Error resolving password  from the credential providers 
	at org.apache.sqoop.util.password.CredentialProviderHelper.resolveAlias(CredentialProviderHelper.java:114)
	at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1065)
	at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
	at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
	at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
	at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
	at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
	at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Configuration problem with provider path.
	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1970)
	at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.sqoop.util.password.CredentialProviderHelper.resolveAlias(CredentialProviderHelper.java:106)
	... 31 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
	at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:867)
	at java.security.KeyStore.load(KeyStore.java:1214)
	at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:110)
	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49)
	at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41)
	at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
	... 37 more

 The exact same sqoop command works via command line though.

sqoop import -Dhadoop.security.credential.provider.path=jceks://hdfs/user/test_user/my.pwd.jceks \
 --connect jdbc:oracle:thin:@//server1.test.com:1526/DB \
 --username test_user\
 --password-alias my.pwd.alias \
 --table MYDB.CUSTOMER \
 --target-dir /user/projects/customer \
 --as-avrodatafile \
 --compression-codec snappy \
 -m 1

   The JKS file is not corrupted if that would have been the real issue the command line wouldn't have worked either.

    Appreciate your feedbak or suggestions. 

[Kumartg]

Hello,

Add the following configuration property to each Sqoop action that uses keystore and re-run the oozie job.

This would resolve the issue - "Error resolving password  from the credential providers "

<property>

<name>oozie.launcher.mapred.map.child.env</name>

<value>HADOOP_CREDSTORE_PASSWORD=none</value>

</property>

Thanks,

Kumar

[omhdfs]

Hello,
Could you please let me know the reason why do we need to add this property?

[ganeshkumarj]

Did you find solution for the problem. I am facing same kind of issue.

[saurabhvora]

Getting following error message

"Error resolving password  from the credential providers"

When trying to execute sqoop import job inside oozie workflow via hue. Can anyone please suggest some solution to it.

[dpugazhe]

 I am trying to distcp operation 

to trasnfer file from ADLS to cluster

hadoop distcp -Dfs.adl.oauth2.access.token.provider.type=ClientCredential-Dhadoop.security.credential.provider.path=jceks://hdfs/adlskeyfilenew.jceks adl://rush.azuredatalakestore.net/src

hdfs:///projects/test

I have provided the HADOOP_CREDSTORE_PASSWORD in command line. here is the error message I am getting

Error: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2048)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1967)
at org.apache.hadoop.fs.adl.AdlFileSystem.getPasswordString(AdlFileSystem.java:993)
at org.apache.hadoop.fs.adl.AdlFileSystem.getConfCredentialBasedTokenProvider(AdlFileSystem.java:278)
at org.apache.hadoop.fs.adl.AdlFileSystem.getAccessTokenProvider(AdlFileSystem.java:257)
at org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:162)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:217)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:335)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:88)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:63)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2028)
... 21 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed

Note : Able to retrive the file . It fails only when i do distcp . Is there any additional command i need to provide for distcp