Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie workflow problem

Highlighted

oozie workflow problem

New Contributor

Hi, I want to call spark from oozie workflow and also delete some files from hdfs and write the new files with same name under the same dir. I put a script into hdfs that execute these commands but I got some errors. I am doing this from HUE UI. The first error is that when i delete the files and rewrite them they are changing permissions and i can't delete them again(permission denied). I managed this by unchecking the option in cloudera manager --> Hdfs check permissions. The second error, that i can't figure it out, is that when the script calls spark i get [Errno13] Permission Denied. The command that script executes is "spark-submit bin/cp_frcast_revenue.py". The file cp_frcast_revenue.py has ugo+rwx permissions. Can you please help me?

1 REPLY 1

Re: oozie workflow problem

New Contributor

Running Oozie (shell script) through Hue I get the follwing error. 

 

The thing is, even if I chown the following file and path to "cloudera" (the user running the script), after some time the file and directory is agaoin chown'd to the user by some script!!!

 



2017-02-07 03:11:52,502 FATAL [main] org.apache.hadoop.conf.Configuration: error parsing conf core-site.xml java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1233) at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:225) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1450) 2017-02-07 03:11:52,505 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.ExceptionInInitializerError at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:225) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1450) Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2650) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1233) at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) ... 5 more Caused by: java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550) ... 10 more 2017-02-07 03:11:52,545 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
Don't have an account?
Coming from Hortonworks? Activate your account here