Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume configurations

Highlighted

Flume configurations

Expert Contributor

Hi,

How to load .xml file into HDFS? just I wanted to know particular configurations to load XML file.

2 REPLIES 2

Re: Flume configurations

Mentor

you have many different choices to load XML into HDFS. You can run simple CLI command

hdfs dfs -put xmlfile.xml /user/directory

you can use Apache Nifi to parse and route to HDFS your XML files

you can use Ambari HDFS view to upload XML files via browser.

Please expand on your use case scenario.

Re: Flume configurations

Expert Contributor

Here is a very simple configuration that uses a Spooling Directory Source, a FileChannel, and an HDFS Sink:

a1.sources = src1
a1.sinks = k1
a1.channels = c1

a1.channels.c1.type = file
a1.sources.src1.type = spooldir
a1.sources.src1.channels = c1
a1.sources.src1.spoolDir = /var/log/apache/flumeSpool

a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S

You can also achieve the same by using Nifi in its most simple form using a GetFile and PutHDFS processor.

Don't have an account?
Coming from Hortonworks? Activate your account here