Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there any way to add workflow feature on nifi dataflow

Highlighted

Is there any way to add workflow feature on nifi dataflow

Explorer

HI

We are working on HDF nifi cluster. We would like to have some workflow feature on HDF. e.g. If I can execute the oozie task after the data was imported to hdfs through nifi.

nifi-dataflow.jpg

Is there any idea to implement the behavior.

Thanks

Paul

5 REPLIES 5

Re: Is there any way to add workflow feature on nifi dataflow

Mentor

You can try using ExecuteProcess or ExecuteScript to call an Oozie job.

Re: Is there any way to add workflow feature on nifi dataflow

Explorer

Hello, @Artem Ervits

Thanks for your quickly response. There is my requirement:

1. To ETL Mysql data to HDFS, the data is increment data.

2. when the data ETL over then call an oozie job to merge data. and so on.

The problem is difficult to check the ETL is over, the increment data are insert to hdfs.

Could you give me advise?

Thanks

Paul

Re: Is there any way to add workflow feature on nifi dataflow

Mentor

For incremental loads from RDBMS, Nifi has QueryDatabaseTable processor, will track updates and inserts to tables but not deletes. If you can live with that limitation, you can get by without Oozie.

Re: Is there any way to add workflow feature on nifi dataflow

Explorer

Hello,@Artem Ervits

Yes, you're right.

However, I would like to know how to check the data flow is over?

It is must to check the status of every components, for me, It is difficult. Could you give me any points?

Thanks

Paul

Re: Is there any way to add workflow feature on nifi dataflow

Mentor

I would leverage REST API in Nifi to do that https://nifi.apache.org/docs/nifi-docs/rest-api/ @Paul Yang

Don't have an account?
Coming from Hortonworks? Activate your account here