Using flume to upload syslog into hdfs on clodera CDH5

Yoavb — Fri, 16 Sep 2022 11:54:22 GMT

Please consider the following scenario :

source : on premise webserver push syslog to the flume
flume :    on cloud , running as a service in cloudera 5.11
target :   on cloud , cloudera hdfs

I need to load data from on-premiss webserver that push syslog into hadoop using flume.
Please note that the flume is running as a service in the CDH

The syslog is sent to the flume server on port 514 (UDP)

The flume agent config file is as follow:

agent.sources = logstream
agent.channels = memoryChannel
agent.sinks = hdfsSink

# I'll be using UDP based Syslog source
agent.sources.logstream.type = syslogudp
# the port that Flume Syslog source will listen on
agent.sources.logstream.port = 514
agent.sources.logstream.host = <flum_server_ip>
agent.channels.memoryChannel.type = memory
agent.sources.logstream.channels = memoryChannel

agent.sinks.hdfsSink.type = logger
agent.sources.logstream.restart = true
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://<masternode>:8020/incoming
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
agent.sinks.hdfsSink.hdfs.filePrefix = %[localhost]-%t-

The stderr show the following error message :

Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogUDPSource{name:logstream,state:IDLE} } - Exception follows.
org.jboss.netty.channel.ChannelException: Failed to bind to: /xxx.xx.xx.xx:514                                                            ======> The xx.xx.xx.xx represent the ip address of the FLUME server
   at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204)
   at org.apache.flume.source.SyslogUDPSource.start(SyslogUDPSource.java:123)
   at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
   at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Permission denied
   at java.net.PlainDatagramSocketImpl.bind0(Native Method)
   at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95)
   at java.net.DatagramSocket.bind(DatagramSocket.java:376)
   at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.bind(OioDatagramPipelineSink.java:89)
   at org.jboss.netty.channel.socket.oio.OioDatagramPipelineSink.eventSunk(OioDatagramPipelineSink.java:60)
   at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
   at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299)
   at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265)
   at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
   at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
   at org.jboss.netty.channel.Channels.bind(Channels.java:561)
   at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
   at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198)
   ... 10 more

Please note that i also tried to set the host address to :
- localhost
- 0.0.0.0
- the port of the webserver
but in all cases i got the error above.

Kindly advise what i am missing here ....

Regards

Re: Using flume to upload syslog into hdfs on clodera CDH5

bgooley — Sun, 09 Jul 2017 16:11:42 GMT

@Yoavb,

Based on the stack, the user running the process does not have permission to bind to port 514 since it is a privileged port. Try using a port higher than 1024. Also web search for "privileged ports" for some background info if needed.

Ben

question Re: Using flume to upload syslog into hdfs on clodera CDH5 in Archives of Support Questions (Read Only)

Using flume to upload syslog into hdfs on clodera CDH5

Re: Using flume to upload syslog into hdfs on clodera CDH5