Automate the process of Pig, Hive, Sqoop.

[#1]

I have data in HDFS(Azure HDInsight) in csv format. I am using Pig to process this Data. After processing the Summarise data will be stored in Hive. And then Hive table is exported in RDBMS using Sqoop. Now I need to automate all this process. Is this possible that I will write particular method for all these 3 task in MapReduce, then run this MapReduce job, and all these task execute one by one.

For create MapReduce job , I want to use .Net SDK. So my question is this possible, and if YES than suggest some steps and reference link for this Question. Thank You.

[#2 · Accepted solution]

@Ishvari Dhimmar

Have you evaluated oozie ? I believe you would need to run these repeatedly at some interval. oozie provides support all the above mentioned components i.e pig , hive and sqoop and can be defined as seperate actions in oozie.

You do not need to create seperate MR job ( using .NET SDK ) if you go this route.

[#3]

Thanks for reply. It will really help me. By mistake I wrote MapReduce Job, I should have to use HiveJob, PigJob, SqoopJob. Thanks again. I just go through Oozie. I didn't find exact Link for Oozie. If I write a Pig script and than want to transfer those data in Hive. Then using Sqoop, export this data to SQL Server. How to connect all these process using Oozie. Can you provide some reference Link?

[#4]

@Ishvari Dhimmar

Please use below links for reference

https://www.tutorialspoint.com/apache_oozie/index.htm

https://oozie.apache.org/docs/4.2.0/DG_Hive2ActionExtension.html

https://oozie.apache.org/docs/4.0.0/DG_SqoopActionExtension.html

https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html