/tmp filling up due to Smartsense files.
SYMPTOMS: /tmp filling up causes multiple services to stop functioning.
ROOT CAUSE: The issue happens due to internal Smartsense bug ST-2551.
SOLUTION: Upgrade to Smartsense 1.3.1
WORKAROUND: To workaround this issue we need to manually modify two files related to Smartsense, so that the tmp files will not be generated in /tmp directory anymore
1. File : /usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py
Change from :
ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\
" -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\ " -cp {1} {2} {3}"
Change to :
ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\
" -Djava.io.tmpdir=/grid/02/smartsense/hst-agent/data/tmp" +\
" -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\
" -cp {1} {2} {3}"
Make sure the tmp dir value is same as this property agent.tmp_dir in hst-agent-conf.
2. File : /usr/sbin/hst-server.py
Change from :
SERVER_START_CMD = "{0}" + os.sep + "bin" + os.sep +\ "java -server -XX:NewRatio=3 "\
"-XX:+UseConcMarkSweepGC " +\
"-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 " +\
debug_options +\
" -Dlog.file.name="+ SERVER_LOG_FILE_NAME +" -Xms512m -Xmx2048m -cp {1}" + os.pathsep + "{2}" +\
" com.hortonworks.support.tools.server.SupportToolServer "\
">" + SERVER_OUT_FILE + " 2>&1 &"
Change to :
SERVER_START_CMD = "{0}" + os.sep + "bin" + os.sep +\
"java -server -XX:NewRatio=3 "\ "-XX:+UseConcMarkSweepGC " +\
"-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 " +\
"-Djava.io.tmpdir=/var/lib/smartsense/hst-server/tmp " +\
debug_options +\
" -Dlog.file.name="+ SERVER_LOG_FILE_NAME +" -Xms512m -Xmx2048m -cp {1}" + os.pathsep + "{2}" +\
" com.hortonworks.support.tools.server.SupportToolServer "\
">" + SERVER_OUT_FILE + " 2>&1 &"
Make sure the tmp dir value is same as this property server.tmp.dir in hst-server-conf.
3. After above changes, pease clean up existing .pyc files from both of the above directories, and restart Smartsense server and agents for changes to take effect.
