Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

custom source in flume not working

custom source in flume not working

Explorer

I am trying to use a custom source in flume based on blog:

http://hadoop-makeitsimple.blogspot.com/2015/05/custom-source-in-flume.html

 

but I see error similar to the post:

http://community.cloudera.com/t5/Data-Ingestion-Integration/Error-in-Configuring-Custom-Flume-Source...

 

 

In next step, I added jar file as described for third  party sources.

 

1] instead of storing the jar file in  /usr/lib/flume-ng/lib, I added it to /var/lib/flume-ng/plugings.d/mydirname/

 

2] I also added class path  in etc/flume-ng/flume-env.sh as:

   FLUME_CLASSPATH= "/var/lib/flume-ng/plugings.d/mydirname"

 

3]  I changed the custom source config as:

a1.sources.r1.type = MySource

a1.sources.r1.classpath = /var/lib/flume-ng/plugings.d/mydirname/*
a1.sources.r1.restart = true
a1.sources.r1.filepath = /home/cloudera/abc/input.txt  -- path to data file

4] command to run config :

 flume-ng agent --name a1 --conf etc/flume-ng/flume-env.sh  -file /home/cloudera/flume_ex/customfile.conf

 

I see the following output:

Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12.jar from classpath
Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-log4j12.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12.jar from classpath
Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12.jar from classpath
...

 ERROR node.Application: A fatal error occurred while running. Exception follows.
org.apache.commons.cli.ParseException: The specified configuration file does not exist: /home/cloudera/ile
	at org.apache.flume.node.Application.main(Application.java:319)

 

I am not sure what I am doing wrong here.

Appreciate the help in advance.

4 REPLIES 4
Highlighted

Re: custom source in flume not working

Explorer

One thing I noticed was using wrong command. I corrected itass :

 

 flume-ng agent --name a1 --conf etc/flume-ng/flume-env.sh  conf-file /home/cloudera/flume_ex/customfile.conf

 

However, I am seeing followign error now:

ERROR node.Application: A fatal error occurred while running. Exception follows.
java.lang.NullPointerException
	at java.io.File.<init>(File.java:277)
	at org.apache.flume.node.Application.main(Application.java:302)

 

I am really stuck here. Can somebody help ?

Highlighted

Re: custom source in flume not working

Super Collaborator
'-file' is not a valid option to start flume-ng. You need to specify --conf and --conf-file (or -f), not -file

Run the 'flume-ng' without any arguments to see the available command line options.

-pd
Highlighted

Re: custom source in flume not working

Explorer

Yes, I corrected in previously replied section. However, I am still getting NullPointerException as in log. Is it related to accessing my custom java source or something else? I am wondering what I am doing wrong here while writing custome java source?

Highlighted

Re: custom source in flume not working

Super Collaborator
If you have a null pointer exception, the stack trace should show you where the line of code is thats throwing the exception, and you should be able to trace it back to your code that isn't properly setting a needed value.

-pd
Don't have an account?
Coming from Hortonworks? Activate your account here