Please consider the following scenario :
source : on premise webserver push syslog to the flume
flume : on cloud , running as a service in cloudera 5.11
target : on cloud , cloudera hdfs
I need to load data from on-premiss webserver that push syslog into hadoop using flume.
Please note that the flume is running as a service in the CDH
The syslog is sent to the flume server on port 514 (UDP)
The flume agent config file is as follow:
agent.sources = logstream
agent.channels = memoryChannel
agent.sinks = hdfsSink
# I'll be using UDP based Syslog source
agent.sources.logstream.type = syslogudp
# the port that Flume Syslog source will listen on
agent.sources.logstream.port = 514
agent.sources.logstream.host = <flum_server_ip>
agent.channels.memoryChannel.type = memory
agent.sources.logstream.channels = memoryChannel
agent.sinks.hdfsSink.type = logger
agent.sources.logstream.restart = true
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t-
The stderr show the following error message :
Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows.
org.jboss.netty.channel.ChannelException: Failed to bind to: /xxx.xx.xx.xx:514 ======> The xx.xx.xx.xx represent the ip address of the FLUME server
at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204)
at org.apache.flume.source.SyslogUDPSource.start(SyslogUDPSource.java:123)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Permission denied
at java.net.PlainDatagramSocketImpl.bind0(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95)
at java.net.DatagramSocket.bind(DatagramSocket.java:376)
at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.bind(OioDatagramPipelineSink.java:89)
at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.eventSunk(OioDatagramPipelineSink.java:60)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299)
at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.jboss.netty.channel.Channels.bind(Channels.java:561)
at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198)
... 10 more
Please note that i also tried to set the host address to :
- localhost
- 0.0.0.0
- the port of the webserver
but in all cases i got the error above.
Kindly advise what i am missing here ....
Regards
