question Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"? in Archives of Support Questions (Read Only)

Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

kbaxley — Wed, 30 Sep 2015 05:00:52 GMT

A customer has a Multi-Nimbus setup. According to the Nimbus HA documentation here:

https://github.com/apache/storm/blob/nimbus-ha-branch/docs/documentation/nimbus-ha-design.md#configuration

...one option for the storm.codedistributor.class is to use "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor".

If this is used, are there any additional configuration settings that need to be put in place for this to be effective?

Re: Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

pbrahmbhatt — Wed, 30 Sep 2015 05:11:17 GMT

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts.

In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future.

Re: Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

soh_dhdv — Wed, 06 Jul 2016 16:33:21 GMT

I don't understand to config HA-mode when i use hdfsbolt. You can give me some inform to load hdfs-site.xml or core-site.xml or other way to understand HA mode.

HdfsBolt bolt = new HdfsBolt() .withFsUrl("hdfs://ha-cluster") .withFileNameFormat(fileNameFormat)

.withRecordFormat(format) .withRotationPolicy(rotationPolicy) .withSyncPolicy(syncPolicy);

When i use .withFsUrl("hdfs://pn2:9000") my topology understand but i used .withFsUrl("hdfs://ha-cluster") is not work.

java.lang.RuntimeException: Error preparing HdfsBolt: java.net.UnknownHostException: ha-cluster
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:109) ~[stormjar.jar:na]
	at backtype.storm.daemon.executor$fn__3439$fn__3451.invoke(executor.clj:699) ~[storm-core-0.9.6.jar:0.9.6]
	at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.6.jar:0.9.6]
	at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: ha-cluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.HdfsBolt.doPrepare(HdfsBolt.java:86) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:105) ~[stormjar.jar:na]
	... 4 common frames omitted