Created 03-10-2017 04:41 PM
my environment requires that I pass
-D ipc.client.fallback-to-simple-auth-allowed=true
to distcp command, in distcp 0.2 action specification for Oozie 4.2, I see java-opts option and I can't seem to make workflow run by passing this property. The only way I can imagine it work is if I put the property in core-site.xml which in production clusters is not feasible. My workflow for reference is
<action name="distcp_1"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <arg>hdfs://aervits-hdp70:8020/tmp/hellounsecure</arg> <arg>hdfs://hacluster:8020/user/centos/</arg> <java-opts>-Dipc.client.fallback-to-simple-auth-allowed=true</java-opts> </distcp> <ok to="end"/> <error to="kill"/> </action>
Created 03-11-2017 12:08 AM
That property is used when one cluster is kerberized and the other isn't, is this your case? If both clusters are kerberized you need to set "oozie.launcher.mapreduce.job.hdfs-servers" property. Regarding your property can you try to pass it as an arg:
<arg>-Dipc.client.fallback-to-simple-auth-allowed=true</arg>
I don't think the space after "-D" matters, but you can try either way.
Created 03-10-2017 04:59 PM
Can you add a space between -D and ipc.client... This is not a Java system property but should be the -D option to the Tool runner
Created 03-10-2017 05:10 PM
@Venkat Ranganathan can you be more specific, I tried with space character, I'm getting
Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid content was found starting with element 'java-opts'. One of '{"uri:oozie:distcp-action:0.2":arg}' is expected.
is this passed as an <arg> or <java-opts>?
<java-opts>-D ipc.client.fallback-to-simple-auth-allowed=true</java-opts>
Created 03-10-2017 05:29 PM
You should have the java-opts right after configuration element and before arg.
Created 03-10-2017 07:17 PM
@Venkat Ranganathan That didn't work, getting
Error: Could not find or load main class ipc.client.fallback-to-simple-auth-allowed=true
in the stderr log. My workflow as of now looks like so
<action name="distcp_1"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <java-opts>-D ipc.client.fallback-to-simple-auth-allowed=true</java-opts> <arg>hdfs://aervits-hdp70:8020/tmp/hellounsecure</arg> <arg>hdfs://hacluster:8020/user/centos/</arg> </distcp> <ok to="end"/> <error to="kill"/> </action>
Created 03-10-2017 08:01 PM
Let me check. The Java opts might be getting added after the class
Created 03-10-2017 10:47 PM
You can set up your args like this and remove from java-opts
<arg>-D</arg> <arg>ipc.client.fallback-to-simple-auth-allowed=true</arg> <arg>hdfs://aervits-hdp70:8020/tmp/hellounsecure</arg> <arg>hdfs://hacluster:8020/user/centos/</arg>
Thanks
Created 03-10-2017 11:15 PM
BTW, this has to come before distcp arguments (like -update etc)
Created 03-11-2017 01:00 AM
@Venkat Ranganathan tried many different options, as part of configuration block as well as a part of submission, i.e.
oozie job -D ipc.client.fallback-to-simple-auth-allowed=true -run
no luck, I read in the Oozie docs that I need property oozie.launcher.mapreduce.job.hdfs-servers and my jobs stopped getting submitted to YARN, hence I commented it out. I also added hadoop.proxy.oozie.hosts and guests to the 2nd cluster as per docs, no luck.
<action name="distcp_1"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <!-- <configuration> <property> <name>oozie.launcher.mapreduce.job.hdfs-servers</name> <value>${nameNode},${nameNode2}</value> </property> </configuration> --> <arg>-D</arg> <arg>ipc.client.fallback-to-simple-auth-allowed=true</arg> <arg>-overwrite</arg> <arg>${nameNode}/user/centos/primary</arg> <arg>${nameNode2}/tmp/</arg> </distcp> <ok to="end"/> <error to="kill"/> </action>
Finally, I stumbled on this note and I'm afraid I hit such case
IMPORTANT: The DistCp action may not work properly with all configurations (secure, insecure) in all versions of Hadoop.
I take instances where one cluster is secured and 2nd cluster is not is not suppored in Distcp action spec 0.2.
https://oozie.apache.org/docs/4.2.0/DG_DistCpActionExtension.html
Created 03-11-2017 12:08 AM
That property is used when one cluster is kerberized and the other isn't, is this your case? If both clusters are kerberized you need to set "oozie.launcher.mapreduce.job.hdfs-servers" property. Regarding your property can you try to pass it as an arg:
<arg>-Dipc.client.fallback-to-simple-auth-allowed=true</arg>
I don't think the space after "-D" matters, but you can try either way.
Created 03-11-2017 02:19 AM
@Predrag Minovic & @Venkat Ranganathan the only way this works with secure and insecure cluster is if core-site.xml has the property below on the secured cluster side.
<property> <name>ipc.client.fallback-to-simple-auth-allowed</name> <value>true</value> </property>
uploading core-site.xml with this property in it in the lib folder of the workflow doesn't work. <arg> and <java-opts> for this property take no effect. job config would not update, the only way to make it work was to update on the hdfs side globally.
Thank you both for your recommendations, eventually it was Predrag who suggested to update HDFS with the property. This is not optimal but for lack of better option will do. It's best to have either two unsecure or two secure clusters.
Created 03-11-2017 02:25 AM
Can you try to put your copy of core-site.xml with that property set to true in the Oozie share lib folder for distcp?
Created 03-11-2017 02:44 AM
adding to sharelib of distcp core-site.xml with this property does not take effect and including it in lib directory of the workflow doesn't take effect either.
Created 03-11-2017 02:57 AM
That's good to know! Many restrictions with Oozie...