Metron Parser Fail to start with Ambari 2.5 and HDP 2.6

[#1]

Hi, I am trying to install Metron Parser in Metron Service in Ambari but it is giving error. Please find below details of stderr.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.2.2.0/package/scripts/parser_master.py", line 93, in <module>
    ParserMaster().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.2.2.0/package/scripts/parser_master.py", line 61, in start
    commands.start_parser_topologies(env)
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.2.2.0/package/scripts/parser_commands.py", line 131, in start_parser_topologies
    user=self.__params.metron_user)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hcp/1.2.2.0-2/metron/bin/start_parser_topology.sh                                     -k sandbox.hortonworks.com:6667                                     -z sandbox.hortonworks.com:2181                                     -s bro                                     -ksp PLAINTEXT' returned 255. Running: /usr/lib/jvm/java/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /usr/hcp/1.2.2.0-2/metron/lib/metron-parsers-0.4.1.1.2.2.0-2-uber.jar /tmp/0516d886954311e7bfe10242ac110002.jar
Running: /usr/lib/jvm/java/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/tmp/0516d886954311e7bfe10242ac110002.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.1.0-129/storm/bin -Dstorm.jar=/tmp/0516d886954311e7bfe10242ac110002.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.metron.parsers.topology.ParserTopologyCLI -k sandbox.hortonworks.com:6667 -z sandbox.hortonworks.com:2181 -s bro -ksp PLAINTEXT
1163 [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
1235 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
1699 [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
1707 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
3111 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7974437777870130871:-5056541602814490343
3190 [main] INFO  o.a.s.u.NimbusClient - Found leader nimbus : sandbox.hortonworks.com:6627
3283 [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
3288 [main] INFO  o.a.s.u.NimbusClient - Found leader nimbus : sandbox.hortonworks.com:6627
3300 [main] INFO  o.a.s.StormSubmitter - Uploading dependencies - jars...
3302 [main] INFO  o.a.s.StormSubmitter - Uploading dependencies - artifacts...
3302 [main] INFO  o.a.s.StormSubmitter - Dependency Blob keys - jars : [] / artifacts : []
3318 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar /tmp/0516d886954311e7bfe10242ac110002.jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-2503662f-08bf-4778-a1de-60f848d48b2d.jar
4118 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-2503662f-08bf-4778-a1de-60f848d48b2d.jar
4118 [main] INFO  o.a.s.StormSubmitter - Submitting topology bro in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7974437777870130871:-5056541602814490343"}
4608 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: bro
4609 [main] INFO  o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
4609 [main] WARN  o.a.s.StormSubmitter - Error occurred in invoking submitter hook:[org.apache.atlas.storm.hook.StormAtlasHook] 
java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_141]
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_141]
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_141]
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_141]
 at java.lang.Class.forName0(Native Method) ~[?:1.8.0_141]
 at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_141]
 at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]
 at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]
 at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]
 at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]
 at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:334) [0516d886954311e7bfe10242ac110002.jar:?]
org.apache.storm.hooks.SubmitterHookException: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook
 at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:368)
 at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:278)
 at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
 at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:162)
 at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:334)
Caused by: java.lang.ClassNotFoundException: org.apache.atlas.storm.hook.StormAtlasHook
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:264)
 at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:361)
 ... 4 more

[#2 · Accepted solution]

Ashikin,

Officially, all versions of Metron, including 1.2.2.0, have not been released to work with anything other then HDP-2.5.x

I am quite sure it does not work with HDP-2.6 since the Storm version is higher, which explains your error message. So you have to revert to a lower version of HDP.

[#3]

@Ashikin Abdullah HCP-1.3 / Metron 0.4.1.1.3.0.0-51 should work with HDP 2.6, but testing is still ongoing, so not for production usage.

[#4]

Alright....Thanks Jasper 🙂

[#5]

@Ashikin Abdullah please mark the question as answered if sufficiently answered.