Support Questions
Find answers, ask questions, and share your expertise

HST server not started

HST server not started

Explorer

stderr:   /var/lib/ambari-agent/data/errors-730.txt

<Failed to execute command:   /usr/sbin/hst start ; Exit code: 1; stdout: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.
Server PID at: /var/run/hst/hst-server.pid
Server out at: /var/log/hst/hst-server.out
Server log at: /var/log/hst/hst-server.log
Waiting for server start....................
User root failed to execute command : /usr/sbin/hst start 
; stderr: Traceback (most recent call last):
  File "/usr/sbin/hst-server.py", line 1557, in <module>
    main(sys.argv)
  File "/usr/sbin/hst-server.py", line 1519, in main
    start_server(options)
  File "/usr/sbin/hst-server.py", line 981, in start_server
    start()
  File "/usr/sbin/hst-server.py", line 885, in start
    raise FatalException(-1, HST_SERVER_DIE_MSG.format(exitcode, get_out_file()))
__main__.FatalException: 'Fatal exception: SmartSense Server java process died with exitcode -1. Check /var/log/hst/hst-server.out for more information., exit code -1'>

stdout:   /var/lib/ambari-agent/data/output-730.txt

<2019-08-26 15:05:37,989 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-26 15:05:38,026 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-26 15:05:38,299 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-26 15:05:38,308 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-26 15:05:38,312 - Group['hdfs'] {}
2019-08-26 15:05:38,313 - Group['hadoop'] {}
2019-08-26 15:05:38,313 - Group['users'] {}
2019-08-26 15:05:38,314 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,316 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,317 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,319 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,320 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,322 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,324 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,326 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,328 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,330 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,331 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,333 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-08-26 15:05:38,335 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,336 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,337 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,338 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,339 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-08-26 15:05:38,352 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-08-26 15:05:38,353 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-08-26 15:05:38,354 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,355 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,355 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2019-08-26 15:05:38,367 - call returned (0, '1015')
2019-08-26 15:05:38,368 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2019-08-26 15:05:38,384 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
2019-08-26 15:05:38,385 - Group['hdfs'] {}
2019-08-26 15:05:38,385 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-08-26 15:05:38,386 - FS Type: HDFS
2019-08-26 15:05:38,386 - Directory['/etc/hadoop'] {'mode': 0755}
2019-08-26 15:05:38,414 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,415 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-08-26 15:05:38,441 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-08-26 15:05:38,449 - Skipping Execute[('setenforce', '0')] due to not_if
2019-08-26 15:05:38,449 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-08-26 15:05:38,451 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-08-26 15:05:38,451 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-08-26 15:05:38,452 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-08-26 15:05:38,456 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-08-26 15:05:38,461 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-08-26 15:05:38,468 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-08-26 15:05:38,485 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,485 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-08-26 15:05:38,486 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,492 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-08-26 15:05:38,501 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-08-26 15:05:38,508 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2019-08-26 15:05:38,823 - The unlimited key JCE policy is required, and appears to have been installed.
2019-08-26 15:05:38,833 - Skipping stack-select on SMARTSENSE because it does not exist in the stack-select package structure.
validating versions
Command: hst --version
Exit code: 0
Std Out: 1.5.1.2.7.3.0-139

Std Err: None
owner for set_permissions:root
Command:  mkdir -p /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
setting permissions: 750
Command:  chmod 750 /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Writing configs to : /etc/hst/conf/hst-agent.ini
Command:  chmod 644 /etc/hst/conf/hst-agent.ini
Exit code: 0
Std Out: None
Std Err: None
fileOwner for config files:
2019-08-26 15:05:39,222 - File['/etc/hst/conf/anonymization_rules.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,225 - File['/etc/hst/conf/productinfo.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,229 - File['/etc/hst/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,236 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-26 15:05:39,236 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json
2019-08-26 15:05:39,237 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json'] {'content': Template('input.config-smartsense.json.j2'), 'mode': 0644}
Writing configs to : /etc/hst/conf/hst-server.ini
Command:  chmod 644 /etc/hst/conf/hst-server.ini
Exit code: 0
Std Out: None
Std Err: None
fileOwner for config files:
2019-08-26 15:05:39,245 - File['/etc/hst/conf/anonymization_rules.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,251 - File['/etc/hst/conf/productinfo.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,253 - File['/etc/hst/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,256 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-26 15:05:39,256 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json
2019-08-26 15:05:39,256 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json'] {'content': Template('input.config-smartsense.json.j2'), 'mode': 0644}
owner for set_permissions:root
Command:  mkdir -p /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
setting permissions: 750
Command:  chmod 750 /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command to be executed: {sudo} /usr/sbin/hst start 
Normalized command:  /usr/sbin/hst start 
Command:   /usr/sbin/hst start 
Exit code: 1
Std Out: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.
Server PID at: /var/run/hst/hst-server.pid
Server out at: /var/log/hst/hst-server.out
Server log at: /var/log/hst/hst-server.log
Waiting for server start....................
User root failed to execute command : /usr/sbin/hst start 

Std Err: Traceback (most recent call last):
  File "/usr/sbin/hst-server.py", line 1557, in <module>
    main(sys.argv)
  File "/usr/sbin/hst-server.py", line 1519, in main
    start_server(options)
  File "/usr/sbin/hst-server.py", line 981, in start_server
    start()
  File "/usr/sbin/hst-server.py", line 885, in start
    raise FatalException(-1, HST_SERVER_DIE_MSG.format(exitcode, get_out_file()))
__main__.FatalException: 'Fatal exception: SmartSense Server java process died with exitcode -1. Check /var/log/hst/hst-server.out for more information., exit code -1'

2019-08-26 15:06:01,426 - Skipping stack-select on SMARTSENSE because it does not exist in the stack-select package structure.

Command failed after 1 tries>
2 REPLIES 2

Re: HST server not started

Hi @Manoj690  ,

 

as mentioned in errorrs-730.txt can you please see whats there in : 

Check /var/log/hst/hst-server.out for more information 

This help you get the correct error on why is it not being started, Please have a look on the out file to know why is it not starting,

 

the file will be present on the node where HST server is installed.

 

Re: HST server not started

Mentor

@Manoj690 

 

Successful Smartsense startup needs an ID to uniquely identifies a customer account. You can obtain it from Cloudera/Hortonworks support. This is a mandatory field during SmartSense installation see below link

 

https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.5.0/reference/content/ss_hst_server.html

 

customer.smartsense.id 

The error in your logs tells you exactly why the server isn't staring

<Failed to execute command: /usr/sbin/hst start ; Exit code: 1; stdout: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.

 

Once you resolve that your smart sense should fire up without any issue.