Reply
New Contributor
Posts: 4
Registered: ‎09-27-2015

oozie workflow problem

Hi, I want to call spark from oozie workflow and also delete some files from hdfs and write the new files with same name under the same dir. I put a script into hdfs that execute these commands but I got some errors. I am doing this from HUE UI. The first error is that when i delete the files and rewrite them they are changing permissions and i can't delete them again(permission denied). I managed this by unchecking the option in cloudera manager --> Hdfs check permissions. The second error, that i can't figure it out, is that when the script calls spark i get [Errno13] Permission Denied. The command that script executes is "spark-submit bin/cp_frcast_revenue.py". The file cp_frcast_revenue.py has ugo+rwx permissions. Can you please help me?

Highlighted
New Contributor
Posts: 1
Registered: ‎02-06-2017

Re: oozie workflow problem

Running Oozie (shell script) through Hue I get the follwing error. 

 

The thing is, even if I chown the following file and path to "cloudera" (the user running the script), after some time the file and directory is agaoin chown'd to the user by some script!!!

 



2017-02-07 03:11:52,502 FATAL [main] org.apache.hadoop.conf.Configuration: error parsing conf core-site.xml java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1233) at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:225) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1450) 2017-02-07 03:11:52,505 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.ExceptionInInitializerError at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:225) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1450) Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2650) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1233) at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) ... 5 more Caused by: java.io.FileNotFoundException: /etc/hadoop/conf.cloudera.yarn/core-site.xml (Permission denied) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550) ... 10 more 2017-02-07 03:11:52,545 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
Announcements