Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

Solved Go to solution
Highlighted

Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

Explorer

A customer has a Multi-Nimbus setup. According to the Nimbus HA documentation here:

https://github.com/apache/storm/blob/nimbus-ha-branch/docs/documentation/nimbus-ha-design.md#configu...

...one option for the storm.codedistributor.class is to use "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor".

If this is used, are there any additional configuration settings that need to be put in place for this to be effective?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

Explorer

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts.

In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future.

View solution in original post

2 REPLIES 2
Highlighted

Re: Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

Explorer

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts.

In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future.

View solution in original post

Highlighted

Re: Any there any additional configurations required after setting "org.apache.storm.hdfs.ha.codedistributor.HDFSCodeDistributor"?

New Contributor

I don't understand to config HA-mode when i use hdfsbolt. You can give me some inform to load hdfs-site.xml or core-site.xml or other way to understand HA mode.

HdfsBolt bolt = new HdfsBolt() .withFsUrl("hdfs://ha-cluster") .withFileNameFormat(fileNameFormat)

.withRecordFormat(format) .withRotationPolicy(rotationPolicy) .withSyncPolicy(syncPolicy);

When i use .withFsUrl("hdfs://pn2:9000") my topology understand but i used .withFsUrl("hdfs://ha-cluster") is not work.

java.lang.RuntimeException: Error preparing HdfsBolt: java.net.UnknownHostException: ha-cluster
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:109) ~[stormjar.jar:na]
	at backtype.storm.daemon.executor$fn__3439$fn__3451.invoke(executor.clj:699) ~[storm-core-0.9.6.jar:0.9.6]
	at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.6.jar:0.9.6]
	at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: ha-cluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) ~[stormjar.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[stormjar.jar:na]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.HdfsBolt.doPrepare(HdfsBolt.java:86) ~[stormjar.jar:na]
	at org.apache.storm.hdfs.bolt.AbstractHdfsBolt.prepare(AbstractHdfsBolt.java:105) ~[stormjar.jar:na]
	... 4 common frames omitted
Don't have an account?
Coming from Hortonworks? Activate your account here