Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3n.

Solved Go to solution

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Yes, it should work, cf. above users using it with no problem. What is your Oozie version?
Highlighted

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

New Contributor
I'm using "oozie-3.3.2-cdh4.5.0". The installation is working fine when input/output paths are specified in form "hdfs://…." but when specified with either "s3://…" or "s3n://" we get the following error in the logs:

Scheme of ʼs3n://...ʼ is not supported.

Where "…" equals path to input/output, verified there, verified working with hadoop commands in console.

Anything you know of I can look at?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

New Contributor
Nevermind, I got it.

The solution was a mix of upgrading the version (to previously listed version) AND adding the supported filesystems property back in.

thanks!

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Glad to hear!

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Master Guru

@Ashok wrote:

My input path in s3n,

 

s3n://xxx-xxx/20130813/08

 

My oozie configuration show as ,

 

hdfs://xxx.internal:8020/s3n://xxx-xxx/20130813/08


Can you share your workflow.xml for us to validate?

 

If you're passing an S3 input or output path, simply ensure your workflow does not template it as ${nameNode}/${input} or something like that. That way you're prepending a HDFS URI to your already-an-uri path. This could most likely be your issue.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Explorer

In coordinator jobs i'm passing the dataset  uri template as 

 

s3n://xxx-xxx/${YEAR}${MONTH}${DAY}/${HOUR}

 

and coord:dataOut as 

 

<property>
<name>in_folder</name>
<value>${coord:dataOut('in_folder')}</value>
</property>

 

and my workflow.xml  input as 

 

${in_folder}

 

when I submit the  coordinator job it automatically preappend  the configuration like:

 

${nameNode}s3n://xxx-xxx/${YEAR}${MONTH}${DAY}/${HOUR}

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Good to know, Hue Coodinators are prepended only with hdfs.

 

Is https://issues.apache.org/jira/browse/OOZIE-426 finished?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Explorer

FWIW, the same job works fine as a workflow when submitted via Hue. In this case, we manually pass the input (S3) and output (hdfs) locations and the job runs successfuly - thus establishing that the problem is not with S3 support. The problem is when we let the co-ordinator pass this input (via a computed datasource) does it automatically prepend hdfs://{nameNode} in front of the s3n://<> URI. Hope this clarifies.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Ok this clarifies a lot! I updated https://issues.cloudera.org/browse/HUE-1501.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Explorer

Thanks. Is this considered as a bug? If yes, what are some workarounds that we can follow for now? Any help is appreciated.

Don't have an account?
Coming from Hortonworks? Activate your account here