Loading data to HDFS - Pig or Spark?

[#1]

Hello,

I want to load some .csv files to HDFS. I already decide that I want to do, in next step, some data transformation with Spark. My question is: I've some advantage to use PIG instead Spark for load data into HDFS?

Many thanks!

[#2 · Accepted solution]

@Johnny Fugers

Have you considered using NiFi to load the data? You can read from many different sources, merge the content into large enough portions to optimize the HDFS use, and write the data directly into HDFS.

[#3]

What I read is a good choice use the same tool for all the steps inside Hadoop. If NiFi gives me that advantages I will study more about it 🙂 Basically, in your opinion I should use:

• NiFi to load data into HDFS
• Spark to do some data transformation (or maybe load data into Hive)

Thanks! 🙂

[#4]

Each of the components within the Hadoop stack has advantages and disadvantages for various tasks. My recommendation is to try to use the best tool for the job while minimizing the amount of complexity you are dealing with. I think if you try to use "the same tool for all the steps" as you read, you may find that your process works, however it may not be optimal.

NiFi is an excellent approach as well, as emaxwell suggested.

[#5]

In my opinion, there are 2 advantages to using Pig for data loading:

1. You are more comfortable programming in Pig.
2. You have existing User Defined Functions (UDFs) that you want to use.

Outside of that, you should be able to what you want to with Spark with faster execution times and arguably much more flexibility. The advantage that Spark provides is the ability to use Java, Scala or Python as the language of choice. You can also you use SQL with Spark, which is something you can't do with Pig.

If you are starting from scratch, give Spark a try.