Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎07-09-2017
Accepted Solution

Using flume to upload syslog into hdfs on clodera CDH5

Please consider the following scenario :

source : on premise webserver push syslog to the flume
flume :    on cloud , running as a service in cloudera 5.11
target :   on cloud , cloudera hdfs

I need to load data from on-premiss webserver that push syslog  into hadoop using flume.
Please note that the flume is running as a service in the CDH

The syslog is sent to the flume server on port 514 (UDP)

The flume agent config file is as follow:

agent.sources = logstream
agent.channels = memoryChannel
agent.sinks = hdfsSink

# I'll be using UDP based Syslog source
agent.sources.logstream.type = syslogudp
# the port that Flume Syslog source will listen on
agent.sources.logstream.port = 514
agent.sources.logstream.host = <flum_server_ip>
agent.channels.memoryChannel.type = memory
agent.sources.logstream.channels = memoryChannel

agent.sinks.hdfsSink.type = logger
agent.sources.logstream.restart = true
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t-

The stderr show the following error message :

Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows.
org.jboss.netty.channel.ChannelException: Failed to bind to: /xxx.xx.xx.xx:514                                                            ======> The xx.xx.xx.xx represent the ip address of the FLUME server
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204)
    at org.apache.flume.source.SyslogUDPSource.start(SyslogUDPSource.java:123)
    at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
    at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Permission denied
    at java.net.PlainDatagramSocketImpl.bind0(Native Method)
    at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95)
    at java.net.DatagramSocket.bind(DatagramSocket.java:376)
    at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.bind(OioDatagramPipelineSink.java:89)
    at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.eventSunk(OioDatagramPipelineSink.java:60)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.bind(Channels.java:561)
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198)
    ... 10 more

Please note that i also tried to set the host address to :
- localhost
- 0.0.0.0
- the port of the webserver
but in all cases i got the error above.

Kindly advise what i am missing here ....

Regards

Posts: 398
Topics: 1
Kudos: 91
Solutions: 51
Registered: ‎04-22-2014

Re: Using flume to upload syslog into hdfs on clodera CDH5

@Yoavb,

 

Based on the stack, the user running the process does not have permission to bind to port 514 since it is a privileged port.  Try using a port higher than 1024.  Also web search for "privileged ports" for some background info if needed.

 

Ben

Announcements