Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to configure the flume.conf file of an agent?

Highlighted

How to configure the flume.conf file of an agent?

Explorer

Hi, I am trying to write the telnet line to my hdfs directory. I have been able to see the telnet lines in the log of my flume agent but i have a different agent that is configured to write the telnet to the hdfs but it does not write it. I am not sure where could be the problem. I am sharing the configuration may be some good genius mind can help me.

This one works as i coud see the telenet lines in the flume-a1.log

apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body: 6D 6F 69 20 6D 6F 69 20 0D moi moi . }

ubuntu@host-192-12-2-81:/etc/flume/conf$ more a1/flume.conf
# Generated by Apache Ambari. Thu Dec 22 11:37:10 2016
a1.channels=c1
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.channels.c1.type=memory
a1.sinks=k1
a1.sinks.k1.channel=c1
a1.sinks.k1.type=logger
a1.sources=r1
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.channels=c1
a1.sources.r1.port=44444
a1.sources.r1.type=netcat

Now the next agent named test configuration which is supposed to put the data into the hdfs. However, i dont see it in hdfs plus i could not find the telnet in the agent log too. I guess that is because i use logger as sinks type.

Here in this second conf file i am critical of two properties first the hdfs path. I tried to put the internal ip 192.168.10.113

of the machine. But i am not sure if the port 8020 is correct. Also about the source port 41414? I am not sure if that is correct. I was previously trying on the port 44445 it did not work and found this 41414 in some hdp manuel. I dont see any error in the agent log which is quite strange.

ubuntu@host-192-12-2-81:/etc/flume/conf$ more test/flume.conf
# Generated by Apache Ambari. Thu Dec 22 11:37:10 2016
test.channels=testchannel
test.channels.testchannel.capacity=1000
test.channels.testchannel.transactionCapacity=100
test.channels.testchannel.type=memory
test.sinks=testsink
test.sinks.testsink.hdfs.filePrefix=alessandro-
test.sinks.testsink.hdfs.path=hdfs://192.168.10.113:8020/data/input
test.sinks.testsink.hdfs.rollCount=1
test.sinks.testsink.hdfs.rollInterval=0
test.sinks.testsink.type=hdfs
test.sources=testsource
test.sources.testsink.channels=testchannel
test.sources.testsource.bind=0.0.0.0
test.sources.testsource.channels=testchannel
test.sources.testsource.port=41414
test.sources.testsource.type=netcat

Your valuable comments are highly appreciated. I really want to see the output in hdfs directories.

2 REPLIES 2

Re: How to configure the flume.conf file of an agent?

Master Collaborator

In case you are still wondering what the issue is in the above, it may be the bad channel config for your sink. Replace the following:

test.sources.testsink.channels=testchannel

with

test.sinks.testsink.channel=testchannel

Re: How to configure the flume.conf file of an agent?

Explorer

yes it was typo with the channel spelling.

Don't have an account?
Coming from Hortonworks? Activate your account here