Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to pass external configuration properties to storm topology?

avatar
Contributor

I want to pass some custom configuration properties to storm topology that are not part of storm yaml, how can I pass it.

1 ACCEPTED SOLUTION

avatar
Super Guru

@Sanjeev Verma

you can use following ways to get the external configuration inside the topplogy

1:

pass the arguments like this

storm jar storm-jar topology-name -c sKey=sValue -c key1=value1 -c key2=value2 >/tmp/storm.txt

2:

Create a simple java resource file (properties files) and pass it as arguments to your topology main class, in main method read the properties from the main file

and build the storm configuration object using conf.put()

3:

create separate yaml file read it through the Utils method provided by storm api,look for more documentation https://nathanmarz.github.io/storm/doc/backtype/storm/utils/Utils.html

Utils.findAndReadConfigFile()

View solution in original post

4 REPLIES 4

avatar
Super Guru

@Sanjeev Verma

you can use following ways to get the external configuration inside the topplogy

1:

pass the arguments like this

storm jar storm-jar topology-name -c sKey=sValue -c key1=value1 -c key2=value2 >/tmp/storm.txt

2:

Create a simple java resource file (properties files) and pass it as arguments to your topology main class, in main method read the properties from the main file

and build the storm configuration object using conf.put()

3:

create separate yaml file read it through the Utils method provided by storm api,look for more documentation https://nathanmarz.github.io/storm/doc/backtype/storm/utils/Utils.html

Utils.findAndReadConfigFile()

avatar

Consider wiring your topologies with Storm's Flux. It also allows for property substitution: http://storm.apache.org/releases/current/flux.html

avatar

@Sanjeev Verma

You can use

Config conf = new Config();

conf.put(key, val);

conf.put(key1, val1);

e.g redis config, etc.

and then you can use this in prepare method of bolts(in case of trident in functions/ filters, etc).

Hope this helps.

avatar
New Contributor

@Andrew Grande, @Rajkumar Singh

I am trying to make the -c work with Flux and the exec-maven-plugin:

call mvn compile exec:java -Dexec.args="--local --sleep 400000 --resource /topology.yaml -c foo.bar=1"

, but I get:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project StorageTopology: An exception occured while executing the Java class. null: InvocationTargetException: Unrecognized option: -c -> [Help 1] 

Does this only work with storm jar submission?

Can I make it work with the exec maven plugin?

Thx,

Mauro.