Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Loading data to HDFS - Pig or Spark?

Solved Go to solution
Highlighted

Loading data to HDFS - Pig or Spark?

New Contributor

Hello,

I want to load some .csv files to HDFS. I already decide that I want to do, in next step, some data transformation with Spark. My question is: I've some advantage to use PIG instead Spark for load data into HDFS?

Many thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Loading data to HDFS - Pig or Spark?

@Johnny Fugers

Have you considered using NiFi to load the data? You can read from many different sources, merge the content into large enough portions to optimize the HDFS use, and write the data directly into HDFS.

4 REPLIES 4

Re: Loading data to HDFS - Pig or Spark?

@Johnny Fugers

Have you considered using NiFi to load the data? You can read from many different sources, merge the content into large enough portions to optimize the HDFS use, and write the data directly into HDFS.

Re: Loading data to HDFS - Pig or Spark?

New Contributor
What I read is a good choice use the same tool for all the steps inside Hadoop. If NiFi gives me that advantages I will study more about it :) Basically, in your opinion I should use:
  • NiFi to load data into HDFS
  • Spark to do some data transformation (or maybe load data into Hive)

Thanks! :)

Re: Loading data to HDFS - Pig or Spark?

Each of the components within the Hadoop stack has advantages and disadvantages for various tasks. My recommendation is to try to use the best tool for the job while minimizing the amount of complexity you are dealing with. I think if you try to use "the same tool for all the steps" as you read, you may find that your process works, however it may not be optimal.

NiFi is an excellent approach as well, as emaxwell suggested.

Re: Loading data to HDFS - Pig or Spark?

In my opinion, there are 2 advantages to using Pig for data loading:

  1. You are more comfortable programming in Pig.
  2. You have existing User Defined Functions (UDFs) that you want to use.

Outside of that, you should be able to what you want to with Spark with faster execution times and arguably much more flexibility. The advantage that Spark provides is the ability to use Java, Scala or Python as the language of choice. You can also you use SQL with Spark, which is something you can't do with Pig.

If you are starting from scratch, give Spark a try.

Don't have an account?
Coming from Hortonworks? Activate your account here