Automate loading data into HDFS

tim_maher4 — Sat, 08 Jul 2017 17:35:14 GMT

hi, I am required to investage setting up a HDFS for our company. Can someone please explain in basic points what we would use to automatically import csv data on a monthly basis, carry out transformations on this data (similar to sql SSIS) and store this data to feed our BI tools?

Any help is very appreciated!

Re: Automate loading data into HDFS

myoung — Mon, 10 Jul 2017 21:49:20 GMT

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

question Re: Automate loading data into HDFS in Archives of Support Questions (Read Only)

Automate loading data into HDFS

Re: Automate loading data into HDFS