Reply
New Contributor
Posts: 5
Registered: ‎05-01-2016

Ozzie job on edgenode

[ Edited ]

We have pyspark program which ingests source files into hdfs and then load them into other hive tables.

Program works fine on edgenode as source files are local in edgenode.Can we schedule this pyspark program through oozie with spark2-submit --deploy-mode client?

Highlighted
Posts: 211
Kudos: 3
Solutions: 4
Registered: ‎08-14-2013

Re: Ozzie job on edgenode

As this is an edge node, Spark 2 is not supported in Oozie actions until CDH 6, and it sounds like these source files may be located specifically on this edgenode, I would suggest using the Oozie SSH action to SSH to this specific edgenode.   You would run a script in the SSH action to run the spark2-submit.  Please see:

 

https://oozie.apache.org/docs/4.1.0/DG_SshActionExtension.html



Robert Justice, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Announcements