Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Automate loading data into HDFS

Solved Go to solution

Automate loading data into HDFS

New Contributor

hi, I am required to investage setting up a HDFS for our company. Can someone please explain in basic points what we would use to automatically import csv data on a monthly basis, carry out transformations on this data (similar to sql SSIS) and store this data to feed our BI tools?

Any help is very appreciated!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Automate loading data into HDFS

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

1 REPLY 1
Highlighted

Re: Automate loading data into HDFS

@Tim Maher

I highly recommend you look into HDF/NiFi as a possible solution to this problem. You can easily use NiFi to pull from source systems, do basic transformations and then store the data in HDFS, Hive, HBase, etc.

In terms of feeding the data to BI tools, you may want to consider storing the data in Hive for the best performance. Take a look at this article for one way to solve some of what you are trying to do: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html