question Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3 in Archives of Support Questions (Read Only)

Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3n.

Ashok — Fri, 16 Sep 2022 08:46:41 GMT

My input path in s3n,

s3n://xxx-xxx/20130813/08

My oozie configuration show as ,

hdfs://xxx.internal:8020/s3n://xxx-xxx/20130813/08

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

dvohra — Tue, 13 Aug 2013 14:06:22 GMT

The link hdfs://xxx.internal:8020/s3n://xxx-xxx/20130813/08 requires a login.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Ashok — Tue, 13 Aug 2013 14:33:04 GMT

Sorry, my question is through Hue in cloudera manager i'm running the oozie job .And I can able to access the hdfs,my question is to connect the another instance Amazon as s3n://xxx to connect ..

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

dvohra — Tue, 13 Aug 2013 16:19:25 GMT

The input path is required to be to HDFS, not S3. S3 is not the same as HDFS.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Harsh J — Sun, 18 Aug 2013 20:39:23 GMT

@dvohra wrote:
The input path is required to be to HDFS, not S3. S3 is not the same as HDFS.

This isn't true. Depending on what you're doing with Oozie, S3 is supported just fine as an input or output location.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Harsh J — Sun, 18 Aug 2013 20:41:11 GMT

@Ashok wrote:
My input path in s3n,

s3n://xxx-xxx/20130813/08

My oozie configuration show as ,

hdfs://xxx.internal:8020/s3n://xxx-xxx/20130813/08

Can you share your workflow.xml for us to validate?

If you're passing an S3 input or output path, simply ensure your workflow does not template it as ${nameNode}/${input} or something like that. That way you're prepending a HDFS URI to your already-an-uri path. This could most likely be your issue.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Ashok — Mon, 19 Aug 2013 06:14:43 GMT

In coordinator jobs i'm passing the dataset uri template as

s3n://xxx-xxx/${YEAR}${MONTH}${DAY}/${HOUR}

and coord:dataOut as

<property>
<name>in_folder</name>
<value>${coord:dataOut('in_folder')}</value>
</property>

and my workflow.xml input as

${in_folder}

when I submit the coordinator job it automatically preappend the configuration like:

${nameNode}s3n://xxx-xxx/${YEAR}${MONTH}${DAY}/${HOUR}

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Romainr — Mon, 19 Aug 2013 06:57:12 GMT

Good to know, Hue Coodinators are prepended only with hdfs.

Is https://issues.apache.org/jira/browse/OOZIE-426 finished?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Ashok — Mon, 19 Aug 2013 08:46:30 GMT

FWIW, the same job works fine as a workflow when submitted via Hue. In this case, we manually pass the input (S3) and output (hdfs) locations and the job runs successfuly - thus establishing that the problem is not with S3 support. The problem is when we let the co-ordinator pass this input (via a computed datasource) does it automatically prepend hdfs://{nameNode} in front of the s3n://<> URI. Hope this clarifies.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Romainr — Mon, 19 Aug 2013 17:28:56 GMT

Ok this clarifies a lot! I updated https://issues.cloudera.org/browse/HUE-1501.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

dvohra — Mon, 19 Aug 2013 22:13:47 GMT

This isn't true. Depending on what you're doing with Oozie, S3 is supported just fine as an input or output location.

Doesn't the coordinator expect the input path to be on HDFS as hdfs://{nameNode} is prepended automatically? The workflow.xml is on the HDFS? Isn't the workflow.xml required to be on the HDFS?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Ashok — Tue, 20 Aug 2013 04:23:45 GMT

Thanks. Is this considered as a bug? If yes, what are some workarounds that we can follow for now? Any help is appreciated.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Harsh J — Wed, 28 Aug 2013 11:09:55 GMT

@dvohra wrote:
This isn't true. Depending on what you're doing with Oozie, S3 is supported just fine as an input or output location.

Doesn't the coordinator expect the input path to be on HDFS as hdfs://{nameNode} is prepended automatically? The workflow.xml is on the HDFS? Isn't the workflow.xml required to be on the HDFS?

Yes unfortunately coordinators currently poll inputs over HDFS alone, which is a limitation. However, writing simple WF actions to work over S3 is still possible.

Yes, WFs should reside on HDFS, as Oozie views it as its central DFS. Similar to how MR requires a proper DFS to run. But this shouldn't impair simple I/O operations done over an external FS such as S3.

I think Romain has covered the relevant JIRAs for tracking removal of this limitation.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

mikeheldar — Mon, 20 Jan 2014 17:05:05 GMT

Is there any updates to this Jira item? Is there a way now to specify an S3n:// location for input or output directories in Oozie workflow without the location being prepended by "hdfs"?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Romainr — Mon, 20 Jan 2014 17:10:13 GMT

Yes, https://issues.cloudera.org/browse/HUE-1501 is in Hue 3.5 or CDH5 beta 2 in one month (and CDH5 in 2 months).

If it is urgent you could try to patch your Hue with the fix: https://issues.cloudera.org/browse/HUE-1501?focusedCommentId=19417&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-19417

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

mikeheldar — Mon, 20 Jan 2014 21:54:09 GMT

Thank you. However I think my question is a little different than that addresses. I am trying to specify an input or output directory in form "s3://..." in an Oozie workflow itself (as an input to a hadoop map reduce job). Do you know if this should work? I get an error that says the path can't have "s3" in it.

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Romainr — Wed, 22 Jan 2014 01:09:12 GMT

Yes, it should work, cf. above users using it with no problem. What is your Oozie version?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

mikeheldar — Thu, 23 Jan 2014 15:31:46 GMT

I'm using "oozie-3.3.2-cdh4.5.0". The installation is working fine when input/output paths are specified in form "hdfs://…." but when specified with either "s3://…" or "s3n://" we get the following error in the logs:

Scheme of ʼs3n://...ʼ is not supported.

Where "…" equals path to input/output, verified there, verified working with hadoop commands in console.

Anything you know of I can look at?

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

mikeheldar — Thu, 23 Jan 2014 15:44:03 GMT

Nevermind, I got it.

The solution was a mix of upgrading the version (to previously listed version) AND adding the supported filesystems property back in.

thanks!

Re: Problem in oozie input path . How do I configure the oozie in cloudera manager,input path as s3

Romainr — Fri, 24 Jan 2014 21:58:15 GMT

Glad to hear!