Automate loading data into HDFS

[#1]

hi, I am required to investage setting up a HDFS for our company. Can someone please explain in basic points what we would use to automatically import csv data on a monthly basis, carry out transformations on this data (similar to sql SSIS) and store this data to feed our BI tools?

Any help is very appreciated!

[#2 · Accepted solution]

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html