Member since
03-11-2017
6
Posts
0
Kudos Received
0
Solutions
03-11-2017
11:59 PM
i am having this error 17/03/11 23:35:34 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 17/03/11 23:35:34 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:flume-agent.properties 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Added sinks: agent-sink Agent: agent 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 WARN conf.FlumeConfiguration: Could not configure sink agent-sink due to: No channel configured for sink: agent-sink org.apache.flume.conf.ConfigurationException: No channel configured for sink: agent-sink at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:51) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:681) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:347) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213) at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127) at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109) at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent] 17/03/11 23:35:34 INFO node.AbstractConfigurationProvider: Creating channels 17/03/11 23:35:34 INFO channel.DefaultChannelFactory: Creating instance of channel agent-chan type memory 17/03/11 23:35:34 INFO node.AbstractConfigurationProvider: Created channel agent-chan 17/03/11 23:35:34 INFO source.DefaultSourceFactory: Creating instance of source agent-src, type spooldir 17/03/11 23:35:34 ERROR node.AbstractConfigurationProvider: Source agent-src has been removed due to an error during configuration java.lang.IllegalStateException: Configuration must specify a spooling directory at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.flume.source.SpoolDirectorySource.configure(SpoolDirectorySource.java:140) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/03/11 23:35:34 WARN node.AbstractConfigurationProvider: Channel agent-chan has no components connected and has been removed. 17/03/11 23:35:34 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} } i made the flume-agent.properties file like this agent.sources = agent-src agent.channels = agent-chan agent.sinks = agent-sink agent.sources.agent-src.type = spooldir agent.sources.agent-src.spooldir = /home/cloudera/desktop/ingestionin agent.sources.agent-src.fileheader = true agent.sources.agent-src.channels = agent-chan agent.channels.agent-chan.type = memory agent.channels.agent-chan.capacity = 10000000 agent.channels.agent-chan.transactioncapacity = 2000 agent.sinks.agent-sink.channels = agent-chan agent.sinks.agent-sink.type = hdfs agent.sinks.agent-sink.writeformat = text agent.sinks.agent-sink.hdfs.filetype = datastream agent.sinks.agent-sink.hdfs.path = new/ingestionout agent.sinks.agent-sink.hdfs.rollcount = 0 agent.sinks.agent-sink.hdfs.rollinterval = 0 agent.sinks.agent-sink.hdfs.rollsize = 0 agent.sinks.agent-sink.hdfs.idletimeout = 0 agent.sinks.agent-sink.hdfs.batchsize = 10000 agent.sinks.agent-sink.hdfs.fileprefix = events but its having errors please let me know whats the error why flume agent is not running. thanks
... View more
Labels:
- Labels:
-
Apache Flume
03-11-2017
11:55 PM
17/03/11 23:35:34 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 17/03/11 23:35:34 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:flume-agent.properties 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Added sinks: agent-sink Agent: agent 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Processing:agent-sink 17/03/11 23:35:34 WARN conf.FlumeConfiguration: Could not configure sink agent-sink due to: No channel configured for sink: agent-sink org.apache.flume.conf.ConfigurationException: No channel configured for sink: agent-sink at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:51) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:681) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:347) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213) at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127) at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109) at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/03/11 23:35:34 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent] 17/03/11 23:35:34 INFO node.AbstractConfigurationProvider: Creating channels 17/03/11 23:35:34 INFO channel.DefaultChannelFactory: Creating instance of channel agent-chan type memory 17/03/11 23:35:34 INFO node.AbstractConfigurationProvider: Created channel agent-chan 17/03/11 23:35:34 INFO source.DefaultSourceFactory: Creating instance of source agent-src, type spooldir 17/03/11 23:35:34 ERROR node.AbstractConfigurationProvider: Source agent-src has been removed due to an error during configuration java.lang.IllegalStateException: Configuration must specify a spooling directory at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.flume.source.SpoolDirectorySource.configure(SpoolDirectorySource.java:140) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/03/11 23:35:34 WARN node.AbstractConfigurationProvider: Channel agent-chan has no components connected and has been removed. 17/03/11 23:35:34 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} } i made the flume-agent.properties file as agent.sources = agent-src agent.channels = agent-chan agent.sinks = agent-sink agent.sources.agent-src.type = spooldir agent.sources.agent-src.spooldir = /home/cloudera/desktop/ingestionin agent.sources.agent-src.fileheader = true agent.sources.agent-src.channels = agent-chan agent.channels.agent-chan.type = memory agent.channels.agent-chan.capacity = 10000000 agent.channels.agent-chan.transactioncapacity = 2000 agent.sinks.agent-sink.channels = agent-chan agent.sinks.agent-sink.type = hdfs agent.sinks.agent-sink.writeformat = text agent.sinks.agent-sink.hdfs.filetype = datastream agent.sinks.agent-sink.hdfs.path = new/ingestionout agent.sinks.agent-sink.hdfs.rollcount = 0 agent.sinks.agent-sink.hdfs.rollinterval = 0 agent.sinks.agent-sink.hdfs.rollsize = 0 agent.sinks.agent-sink.hdfs.idletimeout = 0 agent.sinks.agent-sink.hdfs.batchsize = 10000 agent.sinks.agent-sink.hdfs.fileprefix = events
... View more