Support Questions
Find answers, ask questions, and share your expertise

Automate loading data into HDFS

Solved Go to solution
Highlighted

Automate loading data into HDFS

New Contributor

hi, I am required to investage setting up a HDFS for our company. Can someone please explain in basic points what we would use to automatically import csv data on a monthly basis, carry out transformations on this data (similar to sql SSIS) and store this data to feed our BI tools?

Any help is very appreciated!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Automate loading data into HDFS

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

View solution in original post

1 REPLY 1

Re: Automate loading data into HDFS

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

View solution in original post