question Re: How to pass external configuration properties to storm topology? in Archives of Support Questions (Read Only)

How to pass external configuration properties to storm topology?

sanjeev_verma82 — Wed, 01 Jun 2016 17:40:35 GMT

I want to pass some custom configuration properties to storm topology that are not part of storm yaml, how can I pass it.

Re: How to pass external configuration properties to storm topology?

rajkumar_singh — Wed, 01 Jun 2016 17:50:30 GMT

@Sanjeev Verma

you can use following ways to get the external configuration inside the topplogy

pass the arguments like this

storm jar storm-jar topology-name -c sKey=sValue -c key1=value1 -c key2=value2 >/tmp/storm.txt

Create a simple java resource file (properties files) and pass it as arguments to your topology main class, in main method read the properties from the main file

and build the storm configuration object using conf.put()

create separate yaml file read it through the Utils method provided by storm api,look for more documentation https://nathanmarz.github.io/storm/doc/backtype/storm/utils/Utils.html

Utils.findAndReadConfigFile()

Re: How to pass external configuration properties to storm topology?

andrewg — Wed, 01 Jun 2016 20:43:59 GMT

Consider wiring your topologies with Storm's Flux. It also allows for property substitution: http://storm.apache.org/releases/current/flux.html

Re: How to pass external configuration properties to storm topology?

amber_kulkarni8 — Thu, 03 Nov 2016 18:55:01 GMT

@Sanjeev Verma

You can use

Config conf = new Config();

conf.put(key, val);

conf.put(key1, val1);

e.g redis config, etc.

and then you can use this in prepare method of bolts(in case of trident in functions/ filters, etc).

Hope this helps.

Re: How to pass external configuration properties to storm topology?

maurgi — Thu, 08 Jun 2017 04:08:59 GMT

@Andrew Grande, @Rajkumar Singh

I am trying to make the -c work with Flux and the exec-maven-plugin:

call mvn compile exec:java -Dexec.args="--local --sleep 400000 --resource /topology.yaml -c foo.bar=1"

, but I get:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project StorageTopology: An exception occured while executing the Java class. null: InvocationTargetException: Unrecognized option: -c -> [Help 1]

Does this only work with storm jar submission?

Can I make it work with the exec maven plugin?

Thx,

Mauro.