Support Questions

Find answers, ask questions, and share your expertise

Loading data to HDFS - Pig or Spark?

avatar
Contributor

Hello,

I want to load some .csv files to HDFS. I already decide that I want to do, in next step, some data transformation with Spark. My question is: I've some advantage to use PIG instead Spark for load data into HDFS?

Many thanks!

1 ACCEPTED SOLUTION

avatar

@Johnny Fugers

Have you considered using NiFi to load the data? You can read from many different sources, merge the content into large enough portions to optimize the HDFS use, and write the data directly into HDFS.

View solution in original post

4 REPLIES 4

avatar

@Johnny Fugers

Have you considered using NiFi to load the data? You can read from many different sources, merge the content into large enough portions to optimize the HDFS use, and write the data directly into HDFS.

avatar
Contributor
What I read is a good choice use the same tool for all the steps inside Hadoop. If NiFi gives me that advantages I will study more about it 🙂 Basically, in your opinion I should use:
  • NiFi to load data into HDFS
  • Spark to do some data transformation (or maybe load data into Hive)

Thanks! 🙂

avatar
Super Guru

Each of the components within the Hadoop stack has advantages and disadvantages for various tasks. My recommendation is to try to use the best tool for the job while minimizing the amount of complexity you are dealing with. I think if you try to use "the same tool for all the steps" as you read, you may find that your process works, however it may not be optimal.

NiFi is an excellent approach as well, as emaxwell suggested.

avatar
Super Guru

In my opinion, there are 2 advantages to using Pig for data loading:

  1. You are more comfortable programming in Pig.
  2. You have existing User Defined Functions (UDFs) that you want to use.

Outside of that, you should be able to what you want to with Spark with faster execution times and arguably much more flexibility. The advantage that Spark provides is the ability to use Java, Scala or Python as the language of choice. You can also you use SQL with Spark, which is something you can't do with Pig.

If you are starting from scratch, give Spark a try.