Hive metastore and hiveserver2 won't start

[#1]

running HDP 2.5.3

The root file system filled up on the node that runs hive metastore and hiveserver2. Both processes were stopped. I cleared up disk space but could not get either process to start

hiveserver2 error

yarn.exceptions.ApplicationNotFoundException: Application with id "application_1496410945618_25770" doesn't exist in RM

hivemetastore error

transport exception could not create ServerSocket

I suspect it was a job that didn't finish or die cleanly because of the disk filling up. There were no jobs running in the yarn RM

I tried stopping all the hive processes and restarting yarn and then start hive. Still had the same problem.

I resorted to stopping all the services on the node where the disk filled up and bouncing the node. This fixed the problem as hiveserver2 and hive metastore now run.

Any ideas of a better way to troubleshoot and resolve something like this?

[#2]

For hiveserver2 error:

yarn.exceptions.ApplicationNotFoundException: Application with id "application_1496410945618_25770" doesn't exist in RM

Whenever a application is launched, client will create a directory in the atsactive folder. ATS scans this directory periodically and once the application is done then it moves the application to atsdone directory. There are some scenarios, where stale directory is written in atsactive. ATS keeps scanning those and goes to RM for status of those. RM in return says that it is not aware of that application and ATS then logs there is no application like that. For this message there is already a bug raised:

https://issues.apache.org/jira/browse/YARN-5933

This may not be the cause of hiveserver going down. Could you attach the full log from /var/log/hive? It is better to raise a support case if they are still down.