New Contributor
Posts: 2
Registered: ‎07-09-2017
Accepted Solution

Using flume to upload syslog into hdfs on clodera CDH5

Please consider the following scenario :

source : on premise webserver push syslog to the flume
flume :    on cloud , running as a service in cloudera 5.11
target :   on cloud , cloudera hdfs

I need to load data from on-premiss webserver that push syslog  into hadoop using flume.
Please note that the flume is running as a service in the CDH

The syslog is sent to the flume server on port 514 (UDP)

The flume agent config file is as follow:

agent.sources = logstream
agent.channels = memoryChannel
agent.sinks = hdfsSink

# I'll be using UDP based Syslog source
agent.sources.logstream.type = syslogudp
# the port that Flume Syslog source will listen on
agent.sources.logstream.port = 514 = <flum_server_ip>
agent.channels.memoryChannel.type = memory
agent.sources.logstream.channels = memoryChannel

agent.sinks.hdfsSink.type = logger
agent.sources.logstream.restart = true
agent.sinks.hdfsSink.type = hdfs = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t-

The stderr show the following error message :

Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows. Failed to bind to: /xxx.xx.xx.xx:514                                                            ======> The xx.xx.xx.xx represent the ip address of the FLUME server
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(
    at org.apache.flume.source.SyslogUDPSource.start(
    at org.apache.flume.source.EventDrivenSourceRunner.start(
    at org.apache.flume.lifecycle.LifecycleSupervisor$
    at java.util.concurrent.Executors$
    at java.util.concurrent.FutureTask.runAndReset(
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
    at java.util.concurrent.ScheduledThreadPoolExecutor$
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    at java.util.concurrent.ThreadPoolExecutor$
Caused by: Permission denied
    at Method)
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(
    ... 10 more

Please note that i also tried to set the host address to :
- localhost
- the port of the webserver
but in all cases i got the error above.

Kindly advise what i am missing here ....


Posts: 1,116
Topics: 1
Kudos: 287
Solutions: 135
Registered: ‎04-22-2014

Re: Using flume to upload syslog into hdfs on clodera CDH5



Based on the stack, the user running the process does not have permission to bind to port 514 since it is a privileged port.  Try using a port higher than 1024.  Also web search for "privileged ports" for some background info if needed.