Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to share data between different oozie actions?

SOLVED Go to solution

How to share data between different oozie actions?

Explorer

Hi,

 

I have a requirement where I need to share data between multiple mapreduce actions in a single workflow and between different workflows within single coordinator.

 

I am creating and running my workflows using Hue.

I have read about capture-output element, but this element is not available under mapreduce action. Even if i use Java action to run my mapreduce programs, The capture-element has size constraint of 2KB by default. Could you please let me know how much I can increase the size?

 

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to share data between different oozie actions?

Master Guru
Typically, users use output files on HDFS paths to share data or values between jobs.

The capture-output size is limited cause we store it inside the Oozie DB before we transfer it over to the next action instance. The max size is therefore limited by the size RDBMS supports for CHAR/VARCHAR columns, 64k for MySQL for example.
1 REPLY 1

Re: How to share data between different oozie actions?

Master Guru
Typically, users use output files on HDFS paths to share data or values between jobs.

The capture-output size is limited cause we store it inside the Oozie DB before we transfer it over to the next action instance. The max size is therefore limited by the size RDBMS supports for CHAR/VARCHAR columns, 64k for MySQL for example.