question Re: Data ingestion using flume - Visualize website clickstream data in Archives of Support Questions (Read Only)

Data ingestion using flume - Visualize website clickstream data

Eukrev — Sun, 14 Feb 2016 10:40:39 GMT

Hi,

How can I use Flume to ingest data into HDFS instead of NFS in the below use case? http://hortonworks.com/hadoop-tutorial/how-to-visualize-website-clickstream-data/#section_1

For a semi-structured data where data is placed on Omniture weblog directory, would like to confirm on the source type -

would it be spooling directory source or multiport_syslogtcp. which one to use.

Thank you.

Re: Data ingestion using flume - Visualize website clickstream data

divakarreddy_a — Sun, 14 Feb 2016 11:17:59 GMT

@Revathy Mourouguessane

I hope this link will help for you

https://flume.apache.org/FlumeUserGuide.html

Re: Data ingestion using flume - Visualize website clickstream data

aervits — Sun, 14 Feb 2016 20:48:47 GMT

@Revathy Mourouguessane

spooling dir is good when you want to watch directory for new files. Syslog listens on a port. So if your logs land in a directory, you would use spooling dir. For hdfs you would use hdfs sink. When you master flume, check out Apache NiFi.

Re: Data ingestion using flume - Visualize website clickstream data

Eukrev — Sun, 14 Feb 2016 21:12:49 GMT

Thank you.