- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Apache Oozie Distcp Action with Kerberos enabled clusters
- Labels:
-
Apache Oozie
Created ‎03-10-2017 04:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
my environment requires that I pass
-D ipc.client.fallback-to-simple-auth-allowed=true
to distcp command, in distcp 0.2 action specification for Oozie 4.2, I see java-opts option and I can't seem to make workflow run by passing this property. The only way I can imagine it work is if I put the property in core-site.xml which in production clusters is not feasible. My workflow for reference is
<action name="distcp_1"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <arg>hdfs://aervits-hdp70:8020/tmp/hellounsecure</arg> <arg>hdfs://hacluster:8020/user/centos/</arg> <java-opts>-Dipc.client.fallback-to-simple-auth-allowed=true</java-opts> </distcp> <ok to="end"/> <error to="kill"/> </action>
Created ‎03-11-2017 12:08 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That property is used when one cluster is kerberized and the other isn't, is this your case? If both clusters are kerberized you need to set "oozie.launcher.mapreduce.job.hdfs-servers" property. Regarding your property can you try to pass it as an arg:
<arg>-Dipc.client.fallback-to-simple-auth-allowed=true</arg>
I don't think the space after "-D" matters, but you can try either way.
Created ‎03-11-2017 02:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Predrag Minovic & @Venkat Ranganathan the only way this works with secure and insecure cluster is if core-site.xml has the property below on the secured cluster side.
<property> <name>ipc.client.fallback-to-simple-auth-allowed</name> <value>true</value> </property>
uploading core-site.xml with this property in it in the lib folder of the workflow doesn't work. <arg> and <java-opts> for this property take no effect. job config would not update, the only way to make it work was to update on the hdfs side globally.
Thank you both for your recommendations, eventually it was Predrag who suggested to update HDFS with the property. This is not optimal but for lack of better option will do. It's best to have either two unsecure or two secure clusters.
Created ‎03-11-2017 02:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you try to put your copy of core-site.xml with that property set to true in the Oozie share lib folder for distcp?
Created ‎03-11-2017 02:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
adding to sharelib of distcp core-site.xml with this property does not take effect and including it in lib directory of the workflow doesn't take effect either.
Created ‎03-11-2017 02:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's good to know! Many restrictions with Oozie...

- « Previous
-
- 1
- 2
- Next »