Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎09-19-2016

Second tier Flume agent not binding to first tier

Hi Everyone,

 

I am trying to setup a simple two tier flume agents using the Cloudera VM. 

The code is as follow:

# First-tier agent

agent1.sources = source1 
agent1.sinks = sink1
agent1.channels = channel1

agent1.sources.source1.channels = channel1 
agent1.sinks.sink1.channel = channel1 

agent1.sources.source1.type = spooldir 
agent1.sources.source1.spoolDir = /tmp/spooldir 

agent1.sinks.sink1.type = avro 
agent1.sinks.sink1.hostname = localhost 
agent1.sinks.sink1.port = 10000

agent1.channels.channel1.type = file
agent1.channels.channel1.checkpointDir = /tmp/flume/channel1/check
agent1.channels.channel1.dataDirs = /tmp/flume/channel1/data

# Second-tier agent

agent2.sources = source2 
agent2.sinks = sink2
agent2.channels = channel2

agent2.sources.source2.channels = channel2
agent2.sinks.sink2.channel = channel2

agent2.sources.source2.type = avro 
agent2.sources.source2.bind = localhost 
agent2.sources.source2.port = 10000

agent2.sinks.sink2.type = hdfs 
agent2.sinks.sink2.hdfs.path = /tmp/flume
agent2.sinks.sink2.hdfs.filePrefix = events
agent2.sinks.sink2.hdfs.fileSuffix = .log
agent2.sinks.sink2.hdfs.fileType = DataStream

agent2.channels.channel2.type = file
agent2.channels.channel2.checkpointDir = /tmp/flume/channel2/check
agent2.channels.channel2.dataDirs = /tmp/flume/channel2/data

The first agent looks OK but as soon as I run the second agent, its source cannot connect to the first agent's sink.

It gives the following error: 

17/10/06 14:35:33 INFO node.Application: Starting Sink sink2
17/10/06 14:35:33 INFO node.Application: Starting Source source2
17/10/06 14:35:33 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: sink2: Successfully registered new MBean.
17/10/06 14:35:33 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: sink2 started
17/10/06 14:35:33 INFO source.AvroSource: Starting Avro source source2: { bindAddress: localhost, port: 10000 }...
17/10/06 14:35:35 ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:Avro source source2: { bindAddress: localhost, port: 10000 } } - Exception follows.
org.jboss.netty.channel.ChannelException: Failed to bind to: localhost/127.0.0.1:10000
:10000
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
	at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
	at org.apache.flume.source.AvroSource.start(AvroSource.java:242)
	at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

Does anybbody have any idea why the second flume agent fails to bind? 

 

Thanks.

 

Announcements