Reply
Explorer
Posts: 25
Registered: ‎07-09-2015
Accepted Solution

How to share data between different oozie actions?

Hi,

 

I have a requirement where I need to share data between multiple mapreduce actions in a single workflow and between different workflows within single coordinator.

 

I am creating and running my workflows using Hue.

I have read about capture-output element, but this element is not available under mapreduce action. Even if i use Java action to run my mapreduce programs, The capture-element has size constraint of 2KB by default. Could you please let me know how much I can increase the size?

 

Thanks

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: How to share data between different oozie actions?

Typically, users use output files on HDFS paths to share data or values between jobs.

The capture-output size is limited cause we store it inside the Oozie DB before we transfer it over to the next action instance. The max size is therefore limited by the size RDBMS supports for CHAR/VARCHAR columns, 64k for MySQL for example.
Announcements