Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HST server not started

Highlighted

HST server not started

Explorer

stderr:   /var/lib/ambari-agent/data/errors-730.txt

<Failed to execute command:   /usr/sbin/hst start ; Exit code: 1; stdout: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.
Server PID at: /var/run/hst/hst-server.pid
Server out at: /var/log/hst/hst-server.out
Server log at: /var/log/hst/hst-server.log
Waiting for server start....................
User root failed to execute command : /usr/sbin/hst start 
; stderr: Traceback (most recent call last):
  File "/usr/sbin/hst-server.py", line 1557, in <module>
    main(sys.argv)
  File "/usr/sbin/hst-server.py", line 1519, in main
    start_server(options)
  File "/usr/sbin/hst-server.py", line 981, in start_server
    start()
  File "/usr/sbin/hst-server.py", line 885, in start
    raise FatalException(-1, HST_SERVER_DIE_MSG.format(exitcode, get_out_file()))
__main__.FatalException: 'Fatal exception: SmartSense Server java process died with exitcode -1. Check /var/log/hst/hst-server.out for more information., exit code -1'>

stdout:   /var/lib/ambari-agent/data/output-730.txt

<2019-08-26 15:05:37,989 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-26 15:05:38,026 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-26 15:05:38,299 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-08-26 15:05:38,308 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-08-26 15:05:38,312 - Group['hdfs'] {}
2019-08-26 15:05:38,313 - Group['hadoop'] {}
2019-08-26 15:05:38,313 - Group['users'] {}
2019-08-26 15:05:38,314 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,316 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,317 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,319 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,320 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,322 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,324 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,326 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,328 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,330 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-08-26 15:05:38,331 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,333 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-08-26 15:05:38,335 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,336 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,337 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-08-26 15:05:38,338 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,339 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-08-26 15:05:38,352 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-08-26 15:05:38,353 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-08-26 15:05:38,354 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,355 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-08-26 15:05:38,355 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2019-08-26 15:05:38,367 - call returned (0, '1015')
2019-08-26 15:05:38,368 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2019-08-26 15:05:38,384 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
2019-08-26 15:05:38,385 - Group['hdfs'] {}
2019-08-26 15:05:38,385 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-08-26 15:05:38,386 - FS Type: HDFS
2019-08-26 15:05:38,386 - Directory['/etc/hadoop'] {'mode': 0755}
2019-08-26 15:05:38,414 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,415 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-08-26 15:05:38,441 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-08-26 15:05:38,449 - Skipping Execute[('setenforce', '0')] due to not_if
2019-08-26 15:05:38,449 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-08-26 15:05:38,451 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-08-26 15:05:38,451 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-08-26 15:05:38,452 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-08-26 15:05:38,456 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-08-26 15:05:38,461 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-08-26 15:05:38,468 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-08-26 15:05:38,485 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,485 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-08-26 15:05:38,486 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-08-26 15:05:38,492 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-08-26 15:05:38,501 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-08-26 15:05:38,508 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2019-08-26 15:05:38,823 - The unlimited key JCE policy is required, and appears to have been installed.
2019-08-26 15:05:38,833 - Skipping stack-select on SMARTSENSE because it does not exist in the stack-select package structure.
validating versions
Command: hst --version
Exit code: 0
Std Out: 1.5.1.2.7.3.0-139

Std Err: None
owner for set_permissions:root
Command:  mkdir -p /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
setting permissions: 750
Command:  chmod 750 /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Writing configs to : /etc/hst/conf/hst-agent.ini
Command:  chmod 644 /etc/hst/conf/hst-agent.ini
Exit code: 0
Std Out: None
Std Err: None
fileOwner for config files:
2019-08-26 15:05:39,222 - File['/etc/hst/conf/anonymization_rules.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,225 - File['/etc/hst/conf/productinfo.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,229 - File['/etc/hst/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,236 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-26 15:05:39,236 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json
2019-08-26 15:05:39,237 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json'] {'content': Template('input.config-smartsense.json.j2'), 'mode': 0644}
Writing configs to : /etc/hst/conf/hst-server.ini
Command:  chmod 644 /etc/hst/conf/hst-server.ini
Exit code: 0
Std Out: None
Std Err: None
fileOwner for config files:
2019-08-26 15:05:39,245 - File['/etc/hst/conf/anonymization_rules.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,251 - File['/etc/hst/conf/productinfo.json'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,253 - File['/etc/hst/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': '', 'mode': 0644}
2019-08-26 15:05:39,256 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-08-26 15:05:39,256 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json
2019-08-26 15:05:39,256 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-smartsense.json'] {'content': Template('input.config-smartsense.json.j2'), 'mode': 0644}
owner for set_permissions:root
Command:  mkdir -p /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
setting permissions: 750
Command:  chmod 750 /var/log/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /etc/hst/conf
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/run/hst
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-gateway
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-common
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/data
Exit code: 0
Std Out: None
Std Err: None
Command:  mkdir -p /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command:  chown -R 'root':'root' /var/lib/smartsense/hst-server/tmp
Exit code: 0
Std Out: None
Std Err: None
Command to be executed: {sudo} /usr/sbin/hst start 
Normalized command:  /usr/sbin/hst start 
Command:   /usr/sbin/hst start 
Exit code: 1
Std Out: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.
Server PID at: /var/run/hst/hst-server.pid
Server out at: /var/log/hst/hst-server.out
Server log at: /var/log/hst/hst-server.log
Waiting for server start....................
User root failed to execute command : /usr/sbin/hst start 

Std Err: Traceback (most recent call last):
  File "/usr/sbin/hst-server.py", line 1557, in <module>
    main(sys.argv)
  File "/usr/sbin/hst-server.py", line 1519, in main
    start_server(options)
  File "/usr/sbin/hst-server.py", line 981, in start_server
    start()
  File "/usr/sbin/hst-server.py", line 885, in start
    raise FatalException(-1, HST_SERVER_DIE_MSG.format(exitcode, get_out_file()))
__main__.FatalException: 'Fatal exception: SmartSense Server java process died with exitcode -1. Check /var/log/hst/hst-server.out for more information., exit code -1'

2019-08-26 15:06:01,426 - Skipping stack-select on SMARTSENSE because it does not exist in the stack-select package structure.

Command failed after 1 tries>
2 REPLIES 2

Re: HST server not started

Hi @Manoj690  ,

 

as mentioned in errorrs-730.txt can you please see whats there in : 

Check /var/log/hst/hst-server.out for more information 

This help you get the correct error on why is it not being started, Please have a look on the out file to know why is it not starting,

 

the file will be present on the node where HST server is installed.

 

Re: HST server not started

Mentor

@Manoj690 

 

Successful Smartsense startup needs an ID to uniquely identifies a customer account. You can obtain it from Cloudera/Hortonworks support. This is a mandatory field during SmartSense installation see below link

 

https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.5.0/reference/content/ss_hst_server.html

 

customer.smartsense.id 

The error in your logs tells you exactly why the server isn't staring

<Failed to execute command: /usr/sbin/hst start ; Exit code: 1; stdout: Invalid SmartSense ID: unspecified
ERROR: Validation failed.
Upgrading database...
SmartSense Server is not running
Done upgrading the database.

 

Once you resolve that your smart sense should fire up without any issue.