Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

avatar
Contributor

A customer has a Multi-Nimbus setup. According to the Nimbus HA documentation here:

https://github.com/apache/storm/blob/nimbus-ha-branch/docs/documentation/nimbus-ha-design.md#configu...

...one option for the storm.codedistributor.class is to use "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor".

If this is used, are there any additional configuration settings that need to be put in place for this to be effective?

1 ACCEPTED SOLUTION

avatar
Explorer

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts.

In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future.

View solution in original post

2 REPLIES 2

avatar
Explorer

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts.

In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future.

avatar
New Contributor

I don't understand to config HA-mode when i use hdfsbolt. You can give me some inform to load hdfs-site.xml or core-site.xml or other way to understand HA mode.

HdfsBolt bolt = new HdfsBolt() .withFsUrl("hdfs://ha-cluster") .withFileNameFormat(fileNameFormat)

.withRecordFormat(format) .withRotationPolicy(rotationPolicy) .withSyncPolicy(syncPolicy);

When i use .withFsUrl("hdfs://pn2:9000") my topology understand but i used .withFsUrl("hdfs://ha-cluster") is not work.

java.lang.RuntimeException: Error preparing HdfsBolt: java.net.UnknownHostException: ha-cluster
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:109) ~[stormjar.jar:na]
	at backtype.storm.daemon.executor$fn__3439$fn__3451.invoke(executor.clj:699) ~[storm-core-0.9.6.jar:0.9.6]
	at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.6.jar:0.9.6]
	at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: ha-cluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.HdfsBolt.doPrepare(HdfsBolt.java:86) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:105) ~[stormjar.jar:na]
	... 4 common frames omitted