Created 07-30-2016 08:19 PM
<code>I have installed flume.Running`flume-ng --help` shows that it's installed. After configuring conf file for streaming data from twitter and setting my java path in `flume-env.sh` I'm running the command to stream the data:/bin/flume-ng agent -n TwitterAgent-c conf -f /usr/local/flume/bin/conf /flume.conf
It shows some warnings and some info from log4j. Can any one please help me in this?
<code>Logs: vaibhav@vaibhav-Lenovo-G570:/usr/local/flume/bin$ flume-ng agent -n TwitterAgent-c conf -f /usr/local/flume/conf/flume.conf + exec /usr/local/java/bin/java -Xmx20m-cp 'conf:/usr/local/flume/lib/*'-Djava.library.path= org.apache.flume.node.Application-n TwitterAgent-f /usr/local/flume/conf/flume.conf log4j:WARN No appenders could be found for logger (org.apache.flume.lifecycle.LifecycleSupervisor). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Created 08-01-2016 04:45 AM
Your command is as following, Can you please explain what do you mean by "-f /usr/local/flume" in your command?
$ bin/flume-ng agent -n TwitterAgent -c conf -f /usr/local/flume /conf/flume.conf
Please refer to https://flume.apache.org/FlumeUserGuide.html and look at the argument "-f" to understand what it expects (example: conf/flume-conf.properties.template) :
$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
Created 07-30-2016 11:00 PM
these log4j related warnings are okay to ignore. did you get any other error on console or in logs? if yes then can you please let us know?
Created 07-31-2016 05:13 AM
@Kuldeep Kulkarni thanks for the concern .
Now i get some different error related to java path .i have give the path to jre correctly but it show some . PFA screenshot in the below comment .
vaibhav@vaibhav-Lenovo-G570:/usr/local/flume/bin$ flume-ng agent -n TwitterAgent -c conf -f /usr/local/flume/conf/flume.conf + exec /usr/local/java/jre -Xmx20m -cp 'conf:/usr/local/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -n TwitterAgent -f /usr/local/flume/conf/flume.conf /usr/local/flume/bin/flume-ng: line 241: /usr/local/java/jre: Is a directory /usr/local/flume/bin/flume-ng: line 241: exec: /usr/local/java/jre: cannot execute: Is a directory
Created 07-31-2016 05:17 AM
Created 07-31-2016 06:45 AM
+1 to answer given by @Joy - Can you please check his answer and try to set JAVA_HOME properly ?
Created 07-31-2016 06:05 AM
Looks like your JAVA_HOME is not set correctly. Thats why you are getting the following error:
/usr/local/flume/bin/flume-ng: line 241: exec: /usr/local/java/jre: cannot execute: Is a directory
From your previously added comment i think that in your case the "java" executable is present in the following path: "/usr/local/java/bin/java" Can you try using this java path?
Created 07-31-2016 07:08 AM
i have edited the path i'm getting warning from log4j if i consider this is working then why i cannot see any data in hdfs
Created 07-31-2016 07:09 AM
vaibhav@vaibhav-Lenovo-G570:/usr/local/flume/bin$ flume-ng agent -n TwitterAgent -c conf -f /usr/local/flume /conf/flume.conf + exec /usr/local/java/bin/java -Xmx20m -cp 'conf:/usr/local/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -n TwitterAgent -f /usr/local/flume/conf/flume.conf
log4j:WARN No appenders could be found for logger (org.apache.flume.lifecycle.LifecycleSupervisor).
log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Created 07-31-2016 08:05 AM
@Kuldeep Kulkarni i have made the changes it just showing me the logs mentioned above .No data is going to the created directory.
Created 07-31-2016 09:27 AM
Looks like that your flume is not able to locate the "conf" directory properly because if it will not find the "conf/log4j.properties" then it will complain about the "log4j:WARN No appenders could be found for logger".
In your case as the "conf" directory path is not correct hence it is not able to locate the "log4j.properties" to be added to it's classpath.
Can you try starting the flume as following so that the "conf" directory relative path will be correct.
cd /usr/local/flume ./bin/flume-ng agent -n TwitterAgent -c conf -f /usr/local/flume /conf/flume.conf
As in the above command we are in "/usr/local/flume " directory so when we pass an argument "-c conf" then it should be locate it properly. However in your case you were running the command from inside the "/usr/local/flume/bin/flume-ng -c conf" , So it was trying to find the conf directory inside "/usr/local/flume/bin/"