Per the Storm documentation (http://storm.apache.org/releases/0.10.0/Setting-up-a-Storm-cluster.html): "If you need support from external libraries or custom plugins, you can place such jars into the extlib/ and extlib-daemon/ directories. Note that the extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. Accordingly, two environmental variables STORM_EXT_CLASSPATH and STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the external classpath and daemon-only external classpath." This means that extlib-daemon should not be on the classpath for workers. This functionality was introduced with STORM-483 (https://github.com/apache/storm/commit/05306d5053ff91bd323c4b54cd246c9f928ca339), and supervisor.clj was supposed to be updated as follows: topo-classpath (if-let [cp (storm-conf TOPOLOGY-CLASSPATH)]
[cp]
[])
- classpath (-> (current-classpath)
+ classpath (-> (worker-classpath)
(add-to-classpath [stormjar])
(add-to-classpath topo-classpath))
top-gc-opts (storm-conf TOPOLOGY-WORKER-GC-CHILDOPTS)
However, I decompiled the Hortonworks storm-core jar, and the old version of the code that calls current-classpath still appears: public static final Var const__10 = RT.var((String)"backtype.storm.util", (String)"add-to-classpath");
public static final Var const__11 = RT.var((String)"backtype.storm.util", (String)"current-classpath");
...
v15 = new Object[1];
v16 = stormjar;
stormjar = null;
v15[0] = v16;
v17 = topo_classpath;
topo_classpath = null;
classpath = ((IFn)supervisor$fn__6546.const__10.getRawRoot()).invoke(((IFn)supervisor$fn__6546.const__10.getRawRoot()).invoke(((IFn)supervisor$fn__6546.const__11.getRawRoot()).invoke(), (Object)RT.vector((Object[])v15)), (Object)v17); I believe that worker-classpath was designed to construct a classpath for a worker JVM that does not include daemon-specific locations (like extlib-daemon). However since the HDP version does not call worker-classpath, the worker ends up inheriting the supervisor's classpath via the call to current-classpath. I checked storm-core-0.10.0.2.4.1.1-3.jar which I believe is the latest HDP build, and it still does not call worker-classpath. This seems like a bug.
... View more
