Member since
07-09-2017
2
Posts
0
Kudos Received
0
Solutions
07-09-2017
08:01 AM
Please consider the following scenario : source : on premise webserver push syslog to the flume flume : on cloud , running as a service in cloudera 5.11 target : on cloud , cloudera hdfs I need to load data from on-premiss webserver that push syslog into hadoop using flume. Please note that the flume is running as a service in the CDH The syslog is sent to the flume server on port 514 (UDP) The flume agent config file is as follow: agent.sources = logstream agent.channels = memoryChannel agent.sinks = hdfsSink # I'll be using UDP based Syslog source agent.sources.logstream.type = syslogudp # the port that Flume Syslog source will listen on agent.sources.logstream.port = 514 agent.sources.logstream.host = <flum_server_ip> agent.channels.memoryChannel.type = memory agent.sources.logstream.channels = memoryChannel agent.sinks.hdfsSink.type = logger agent.sources.logstream.restart = true agent.sinks.hdfsSink.type = hdfs agent.sinks.hdfsSink.channel = memoryChannel agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming agent.sinks.hdfsSink.hdfs.fileType = DataStream agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t- The stderr show the following error message : Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows. org.jboss.netty.channel.ChannelException: Failed to bind to: /xxx.xx.xx.xx:514 ======> The xx.xx.xx.xx represent the ip address of the FLUME server at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) at org.apache.flume.source.SyslogUDPSource.start(SyslogUDPSource.java:123) at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.BindException: Permission denied at java.net.PlainDatagramSocketImpl.bind0(Native Method) at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95) at java.net.DatagramSocket.bind(DatagramSocket.java:376) at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.bind(OioDatagramPipelineSink.java:89) at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.eventSunk(OioDatagramPipelineSink.java:60) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.bind(Channels.java:561) at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189) at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ... 10 more Please note that i also tried to set the host address to : - localhost - 0.0.0.0 - the port of the webserver but in all cases i got the error above. Kindly advise what i am missing here .... Regards
... View more
Labels:
- Labels:
-
Apache Flume
-
Cloudera Manager