Created 07-31-2017 04:59 PM
running HDP 2.5.3
The root file system filled up on the node that runs hive metastore and hiveserver2. Both processes were stopped. I cleared up disk space but could not get either process to start
hiveserver2 error
yarn.exceptions.ApplicationNotFoundException: Application with id "application_1496410945618_25770" doesn't exist in RM
hivemetastore error
transport exception could not create ServerSocket
I suspect it was a job that didn't finish or die cleanly because of the disk filling up. There were no jobs running in the yarn RM
I tried stopping all the hive processes and restarting yarn and then start hive. Still had the same problem.
I resorted to stopping all the services on the node where the disk filled up and bouncing the node. This fixed the problem as hiveserver2 and hive metastore now run.
Any ideas of a better way to troubleshoot and resolve something like this?
Created 08-03-2017 06:30 PM
For hiveserver2 error:
yarn.exceptions.ApplicationNotFoundException: Application with id "application_1496410945618_25770" doesn't exist in RM
Whenever a application is launched, client will create a directory in the atsactive folder. ATS scans this directory periodically and once the application is done then it moves the application to atsdone directory. There are some scenarios, where stale directory is written in atsactive. ATS keeps scanning those and goes to RM for status of those. RM in return says that it is not aware of that application and ATS then logs there is no application like that. For this message there is already a bug raised:
https://issues.apache.org/jira/browse/YARN-5933
This may not be the cause of hiveserver going down. Could you attach the full log from /var/log/hive? It is better to raise a support case if they are still down.