Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. flume

ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. flume

New Contributor

Hi All,
I am running haoop on Ubuntu and trying to get twitter feeds from Flume, I am getting the following error:


==========================================================================
14/03/16 18:42:17 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume/conf/flume.conf
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Processing:HDFS
14/03/16 18:42:17 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent]
14/03/16 18:42:17 INFO node.AbstractConfigurationProvider: Creating channels
14/03/16 18:42:17 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory
14/03/16 18:42:17 INFO node.AbstractConfigurationProvider: Created channel MemChannel
14/03/16 18:42:17 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource
14/03/16 18:42:17 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)
==========================================================================

The details of the files are as below:

==========================================================================
Flume.conf
==========================================================================
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

TwitterAgent.sources=Twitter
TwitterAgent.channels=MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type=com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels=MemChannel

TwitterAgent.sources.Twitter.consumerKey=Pw63cpjptT59ulP0zmT6w
TwitterAgent.sources.Twitter.consumerSecret=n8awrhwk0ui5WOwr3NLKf7S576DcILPk5Ddfp1LQUU
TwitterAgent.sources.Twitter.accessToken=1635433267-s0NAOXmRqm5y4UC2WV7HPOuiOE9fPZZ56eWO95P
TwitterAgent.sources.Twitter.accessTokenSecret=    CBKPgbJLwyJJ1jY4atf7iaiaR96Z1PmVvKF0iOXsP8E

TwitterAgent.sources.Twitter.keywords= priyanka, hadoop, scientist

TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/flume/tweets
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=100

==========================================================================

Flume-env.sh
==========================================================================

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.

JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64


# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH="/usr/lib/flume/lib/flume-sources-1.0-SNAPSHOT.jar"

==========================================================================

 

Please help me to solve this..

 

Thanks in Advance,

Thanks,

Krishna Prasad

2 REPLIES 2

Re: ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. flume

Master Guru
Does your jar at /usr/lib/flume/lib/flume-sources-1.0-SNAPSHOT.jar contain the class you require: com.cloudera.flume.source.TwitterSource?

Re: ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. flume

Explorer

 

Hi Harsha,

 

I have some similar kind of problem.I posted with subject name :Flume Twitter analysis

Need your guidence.

 

Thanks

Vishal

Don't have an account?
Coming from Hortonworks? Activate your account here