How to pass external configuration properties to storm topology?

[#1]

I want to pass some custom configuration properties to storm topology that are not part of storm yaml, how can I pass it.

[#2 · Accepted solution]

@Sanjeev Verma

you can use following ways to get the external configuration inside the topplogy

1:

pass the arguments like this

storm jar storm-jar topology-name -c sKey=sValue -c key1=value1 -c key2=value2 >/tmp/storm.txt

2:

Create a simple java resource file (properties files) and pass it as arguments to your topology main class, in main method read the properties from the main file

and build the storm configuration object using conf.put()

3:

create separate yaml file read it through the Utils method provided by storm api,look for more documentation https://nathanmarz.github.io/storm/doc/backtype/storm/utils/Utils.html

Utils.findAndReadConfigFile()

[#3]

Consider wiring your topologies with Storm's Flux. It also allows for property substitution: http://storm.apache.org/releases/current/flux.html

[#4]

@Sanjeev Verma

You can use

Config conf = new Config();

conf.put(key, val);

conf.put(key1, val1);

e.g redis config, etc.

and then you can use this in prepare method of bolts(in case of trident in functions/ filters, etc).

Hope this helps.

[#5]

@Andrew Grande, @Rajkumar Singh

I am trying to make the -c work with Flux and the exec-maven-plugin:

call mvn compile exec:java -Dexec.args="--local --sleep 400000 --resource /topology.yaml -c foo.bar=1"

, but I get:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project StorageTopology: An exception occured while executing the Java class. null: InvocationTargetException: Unrecognized option: -c -> [Help 1] 

Does this only work with storm jar submission?

Can I make it work with the exec maven plugin?

Thx,

Mauro.