Metron REST cannot login using HDP 2.5 with HCP 1.3

[#1]

Hi, I have a problem. There are no alert (green tick) but I cannot start Metron REST Service in Ambari and login into Metron Management UI, but there is no standard error though. Below is the details of stdout. There are also error in /var/log/metron/metron-rest.log. Thanks.

/var/log/metron/metron-rest.log

Error: Could not find or load main class org.apache.metron.rest.MetronRestApplication

stderr: /var/lib/ambari-agent/data/errors-2944.txt

None

stdout: /var/lib/ambari-agent/data/output-2944.txt

2017-11-14 19:28:47,209 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37
2017-11-14 19:28:47,222 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-14 19:28:47,401 - Stack Feature Version Info: Cluster Stack=2.5, Cluster Current Version=None, Command Stack=None, Command Version=2.5.3.0-37 -> 2.5.3.0-37
2017-11-14 19:28:47,408 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-11-14 19:28:47,409 - Group['metron'] {}
2017-11-14 19:28:47,410 - Group['elasticsearch'] {}
2017-11-14 19:28:47,410 - Group['zeppelin'] {}
2017-11-14 19:28:47,410 - Group['hadoop'] {}
2017-11-14 19:28:47,410 - Group['kibana'] {}
2017-11-14 19:28:47,410 - Group['users'] {}
2017-11-14 19:28:47,411 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,412 - call['/var/lib/ambari-agent/tmp/changeUid.sh hive'] {}
2017-11-14 19:28:47,423 - call returned (0, '1001')
2017-11-14 19:28:47,423 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1001}
2017-11-14 19:28:47,424 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,425 - call['/var/lib/ambari-agent/tmp/changeUid.sh storm'] {}
2017-11-14 19:28:47,435 - call returned (0, '1002')
2017-11-14 19:28:47,436 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1002}
2017-11-14 19:28:47,437 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,437 - call['/var/lib/ambari-agent/tmp/changeUid.sh zookeeper'] {}
2017-11-14 19:28:47,448 - call returned (0, '1003')
2017-11-14 19:28:47,449 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1003}
2017-11-14 19:28:47,451 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,452 - call['/var/lib/ambari-agent/tmp/changeUid.sh ams'] {}
2017-11-14 19:28:47,462 - call returned (0, '1004')
2017-11-14 19:28:47,462 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1004}
2017-11-14 19:28:47,464 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,464 - call['/var/lib/ambari-agent/tmp/changeUid.sh tez'] {}
2017-11-14 19:28:47,476 - call returned (0, '1005')
2017-11-14 19:28:47,477 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': 1005}
2017-11-14 19:28:47,478 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,478 - call['/var/lib/ambari-agent/tmp/changeUid.sh zeppelin'] {}
2017-11-14 19:28:47,487 - call returned (0, '1007')
2017-11-14 19:28:47,488 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop'], 'uid': 1007}
2017-11-14 19:28:47,490 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,491 - call['/var/lib/ambari-agent/tmp/changeUid.sh metron'] {}
2017-11-14 19:28:47,500 - call returned (0, '1008')
2017-11-14 19:28:47,501 - User['metron'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1008}
2017-11-14 19:28:47,502 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,504 - call['/var/lib/ambari-agent/tmp/changeUid.sh elasticsearch'] {}
2017-11-14 19:28:47,512 - call returned (0, '1010')
2017-11-14 19:28:47,513 - User['elasticsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1010}
2017-11-14 19:28:47,514 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2017-11-14 19:28:47,516 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,517 - call['/var/lib/ambari-agent/tmp/changeUid.sh flume'] {}
2017-11-14 19:28:47,526 - call returned (0, '1012')
2017-11-14 19:28:47,527 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1012}
2017-11-14 19:28:47,528 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,529 - call['/var/lib/ambari-agent/tmp/changeUid.sh kafka'] {}
2017-11-14 19:28:47,539 - call returned (0, '1013')
2017-11-14 19:28:47,539 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1013}
2017-11-14 19:28:47,541 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,542 - call['/var/lib/ambari-agent/tmp/changeUid.sh hdfs'] {}
2017-11-14 19:28:47,551 - call returned (0, '1014')
2017-11-14 19:28:47,551 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1014}
2017-11-14 19:28:47,553 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,555 - call['/var/lib/ambari-agent/tmp/changeUid.sh yarn'] {}
2017-11-14 19:28:47,564 - call returned (0, '1015')
2017-11-14 19:28:47,564 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1015}
2017-11-14 19:28:47,566 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,567 - call['/var/lib/ambari-agent/tmp/changeUid.sh kibana'] {}
2017-11-14 19:28:47,576 - call returned (0, '1016')
2017-11-14 19:28:47,577 - User['kibana'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1016}
2017-11-14 19:28:47,578 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,579 - call['/var/lib/ambari-agent/tmp/changeUid.sh mapred'] {}
2017-11-14 19:28:47,588 - call returned (0, '1017')
2017-11-14 19:28:47,588 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1017}
2017-11-14 19:28:47,590 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,591 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2017-11-14 19:28:47,599 - call returned (0, '1018')
2017-11-14 19:28:47,600 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1018}
2017-11-14 19:28:47,601 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,602 - call['/var/lib/ambari-agent/tmp/changeUid.sh hcat'] {}
2017-11-14 19:28:47,613 - call returned (0, '1019')
2017-11-14 19:28:47,614 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': 1019}
2017-11-14 19:28:47,615 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,617 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-11-14 19:28:47,623 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2017-11-14 19:28:47,623 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2017-11-14 19:28:47,624 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,626 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-11-14 19:28:47,627 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2017-11-14 19:28:47,637 - call returned (0, '1018')
2017-11-14 19:28:47,637 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1018'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2017-11-14 19:28:47,643 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1018'] due to not_if
2017-11-14 19:28:47,644 - Group['hdfs'] {}
2017-11-14 19:28:47,644 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2017-11-14 19:28:47,645 - FS Type: 
2017-11-14 19:28:47,645 - Directory['/etc/hadoop'] {'mode': 0755}
2017-11-14 19:28:47,663 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2017-11-14 19:28:47,664 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-11-14 19:28:47,683 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-11-14 19:28:47,696 - Skipping Execute[('setenforce', '0')] due to only_if
2017-11-14 19:28:47,697 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2017-11-14 19:28:47,700 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2017-11-14 19:28:47,701 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2017-11-14 19:28:47,708 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2017-11-14 19:28:47,711 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2017-11-14 19:28:47,721 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-11-14 19:28:47,735 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-11-14 19:28:47,736 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-11-14 19:28:47,737 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-11-14 19:28:47,743 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2017-11-14 19:28:47,749 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-11-14 19:28:47,990 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-14 19:28:48,001 - File['/etc/default/metron'] {'content': Template('metron.j2')}
2017-11-14 19:28:48,002 - Directory['/var/run/metron'] {'owner': 'metron', 'group': 'metron', 'create_parents': True, 'mode': 0755}
2017-11-14 19:28:48,003 - Directory['/var/log/metron'] {'owner': 'metron', 'group': 'metron', 'create_parents': True, 'mode': 0755}
2017-11-14 19:28:48,003 - Creating Kafka topics for rest
2017-11-14 19:28:48,003 - Creating Kafka topics
2017-11-14 19:28:48,003 - Creating topic'escalation'
2017-11-14 19:28:48,003 - Execute['/usr/hdp/current/kafka-broker/bin/kafka-topics.sh                             --zookeeper slot4:2181,slot3:2181,slot2:2181                             --create                             --if-not-exists                             --topic escalation                             --partitions 1                             --replication-factor 1                             --config retention.bytes=10737418240'] {'logoutput': True, 'tries': 3, 'user': 'kafka', 'try_sleep': 5}
2017-11-14 19:28:49,680 - Done creating Kafka topics
2017-11-14 19:28:49,681 - Directory['/var/run/metron'] {'owner': 'metron', 'group': 'metron', 'create_parents': True, 'mode': 0755}
2017-11-14 19:28:49,682 - Directory['/var/log/metron'] {'owner': 'metron', 'group': 'metron', 'create_parents': True, 'mode': 0755}
2017-11-14 19:28:49,682 - Starting REST application
2017-11-14 19:28:49,684 - call['ambari-sudo.sh su metron -l -s /bin/bash -c 'cat /var/run/metron/metron-rest.pid 1>/tmp/tmpFx5rE5 2>/tmp/tmpp9HGqu''] {'quiet': False}
2017-11-14 19:28:49,893 - call returned (0, '')
2017-11-14 19:28:49,895 - Execute['set -o allexport; source /etc/default/metron; set +o allexport;export METRON_JDBC_PASSWORD=[PROTECTED];/usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hbase-client/conf:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0.jar:/usr/share/java/mysql-connector-java.jar:/usr/hcp/1.3.0.0-51/metron/lib/metron-elasticsearch-0.4.1.1.3.0.0-uber.jar org.apache.metron.rest.MetronRestApplication  --server.port=8082 --spring.config.location=/usr/hcp/1.3.0.0-51/metron/config/rest_application.yml >> /var/log/metron/metron-rest.log 2>&1 & echo $! > /var/run/metron/metron-rest.pid;unset METRON_JDBC_PASSWORD;'] {'logoutput': True, 'not_if': 'ls /var/run/metron/metron-rest.pid >/dev/null 2>&1 && ps -p 33489 >/dev/null 2>&1', 'user': 'metron'}
2017-11-14 19:28:50,115 - Done starting REST application

Command completed successfully!

Here's from Metron REST config tab

[#2 · Accepted solution]

Hello @Ashikin, please try the following and see if it resolves your issue.

In the Ambari UI, go to Services -> Metron -> Configs

Under the 'Filters' search for the parameter metron_indexing_classpath and add the following value:

/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hbase-client/conf:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar:/usr/hcp/1.3.0.0-51/metron/lib/metron-elasticsearch-0.4.1.1.3.0.0-51-uber.jar

Then restart Metron REST and Management UI services

[#3]

Hi @asubramanian, it works like a charm! Thank you so much 🙂

[#4]

Cool @Ashikin!

[#5]

There is also a metron_rest_classpath parameter that can be used for the same purpose (both of these parameters are added to the REST application classpath). You might want to consider keeping just /path/to/metron-elasticsearch-*-uber.jar in metron_indexing_classpath and moving the others to metron_rest_classpath. Probably won't matter now but could cause issues down the road if metron_indexing_classpath is ever reused for something else.