turn on suggestions Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Do you mean 
Reply
Highlighted
M123
Explorer
M123
Posts: 12
Registered: ‎05-30-2017

trying to ingest data using flume from a website to cloudera hdfs directory

‎07-08-2017 09:16 AM

how to deal with this error? Is the channel full?

clienta.properties

 

julyLog.sources= src1
julyLog.channels= chnl1
julyLog.sinks= snk1


#sources
julyLog.sources.src1.type=spooldir
julyLog.sources.src1.channels=chnl1
julyLog.sources.src1.spoolDir=/home/cloudera/sample-data/nasa_log/july
julyLog.sources.src1.inputCharset=us-ascii

#channels
julyLog.channels.chnl1.type=memory
julyLog.channels.chnl1.capacity=10000
julyLog.channels.chnl1.transactionCapacity=1000

#sink
julyLog.sinks.snk1.type = avro
julyLog.sinks.snk1.channel = chnl1
julyLog.sinks.snk1.hostname = localhost
julyLog.sinks.snk1.port = 4545

 

 

 

 

below is the error

 

17/07/08 08:27:27 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:27:27 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:27:28 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:32 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:35 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:39 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:42 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:46 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:47 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:27:49 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:27:52 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:27:52 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:27:52 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:27:53 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:27:56 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:00 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:03 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:07 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:10 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:12 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:28:14 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:17 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:17 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:28:17 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:28:17 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:28:21 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:24 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:28 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:31 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:35 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:37 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:28:38 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:42 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:42 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:28:42 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:28:42 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:28:45 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:49 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:52 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:28:56 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:28:59 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:02 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:29:03 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:06 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:07 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:29:07 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:29:07 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:29:10 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:13 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:17 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:20 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:24 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:27 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:27 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
	... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Handshake timed out after 20000ms
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
	... 4 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:201)
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
	... 5 more
17/07/08 08:29:31 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:32 INFO sink.AbstractRpcSink: Rpc sink snk1: Building RpcClient with hostname: localhost, port: 4545
17/07/08 08:29:32 INFO sink.AvroSink: Attempting to create Avro Rpc client.
17/07/08 08:29:32 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
17/07/08 08:29:34 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:38 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:41 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:45 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:48 WARN source.SpoolDirectorySource: The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
17/07/08 08:29:52 INFO avro.ReliableSpoolingFileEventReader: Last read was never committed - resetting mark position.
17/07/08 08:29:52 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 4545 }: Failed to send batch
	at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
	at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)

 

 

Report Inappropriate Content
Message 1 of 1 (294 Views)
Labels:
Reply
0 Kudos
Announcements

Share Your Thoughts

New solutions
Find More Solutions