Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Using flume to upload syslog into hdfs on clodera CDH5

avatar
New Contributor

Please consider the following scenario :

source : on premise webserver push syslog to the flume
flume :    on cloud , running as a service in cloudera 5.11
target :   on cloud , cloudera hdfs

I need to load data from on-premiss webserver that push syslog  into hadoop using flume.
Please note that the flume is running as a service in the CDH

The syslog is sent to the flume server on port 514 (UDP)

The flume agent config file is as follow:

agent.sources = logstream
agent.channels = memoryChannel
agent.sinks = hdfsSink

# I'll be using UDP based Syslog source
agent.sources.logstream.type = syslogudp
# the port that Flume Syslog source will listen on
agent.sources.logstream.port = 514
agent.sources.logstream.host = <flum_server_ip>
agent.channels.memoryChannel.type = memory
agent.sources.logstream.channels = memoryChannel

agent.sinks.hdfsSink.type = logger
agent.sources.logstream.restart = true
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t-

The stderr show the following error message :

Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows.
org.jboss.netty.channel.ChannelException: Failed to bind to: /xxx.xx.xx.xx:514                                                            ======> The xx.xx.xx.xx represent the ip address of the FLUME server
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204)
    at org.apache.flume.source.SyslogUDPSource.start(SyslogUDPSource.java:123)
    at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
    at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Permission denied
    at java.net.PlainDatagramSocketImpl.bind0(Native Method)
    at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95)
    at java.net.DatagramSocket.bind(DatagramSocket.java:376)
    at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.bind(OioDatagramPipelineSink.java:89)
    at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.eventSunk(OioDatagramPipelineSink.java:60)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.bind(Channels.java:561)
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198)
    ... 10 more

Please note that i also tried to set the host address to :
- localhost
- 0.0.0.0
- the port of the webserver
but in all cases i got the error above.

Kindly advise what i am missing here ....

Regards

1 ACCEPTED SOLUTION

avatar
Master Guru

@Yoavb,

 

Based on the stack, the user running the process does not have permission to bind to port 514 since it is a privileged port.  Try using a port higher than 1024.  Also web search for "privileged ports" for some background info if needed.

 

Ben

View solution in original post

1 REPLY 1

avatar
Master Guru

@Yoavb,

 

Based on the stack, the user running the process does not have permission to bind to port 514 since it is a privileged port.  Try using a port higher than 1024.  Also web search for "privileged ports" for some background info if needed.

 

Ben