Reply
Explorer
Posts: 12
Registered: ‎05-30-2017

trying to ingest data using flume from a website to cloudera hdfs directory

how to deal with this error? Is the channel full?

clienta.properties

 

julyLog.sources= src1
julyLog.channels= chnl1
julyLog.sinks= snk1


#sources
julyLog.sources.src1.type=spooldir
julyLog.sources.src1.channels=chnl1
julyLog.sources.src1.spoolDir=/home/cloudera/sample-data/nasa_log/july
julyLog.sources.src1.inputCharset=us-ascii

#channels
julyLog.channels.chnl1.type=memory
julyLog.channels.chnl1.capacity=10000
julyLog.channels.chnl1.transactionCapacity=1000

#sink
julyLog.sinks.snk1.type = avro
julyLog.sinks.snk1.channel = chnl1
julyLog.sinks.snk1.hostname = localhost
julyLog.sinks.snk1.port = 4545

 

 

 

 

below is the error

 

17/07/08 08:27:27 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:27:27 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:27:28 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:32 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:35 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:39 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:42 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:46 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:47 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:27:49 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:52 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:27:52 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:27:52 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:27:53 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:56 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:00 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:03 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:07 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:10 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:12 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:28:14 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:17 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:17 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:28:17 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:28:17 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:28:21 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:24 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:28 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:31 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:35 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:37 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:28:38 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:42 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:42 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:28:42 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:28:42 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:28:45 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:49 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:52 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:56 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:59 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:02 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:29:03 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:06 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:07 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:29:07 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:29:07 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:29:10 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:13 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:17 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:20 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:24 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:27 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:27 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:29:31 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:32 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:29:32 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:29:32 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:29:34 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:38 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:41 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:45 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:48 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:52 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:52 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)

 

 

Announcements
New solutions