The hive query which is used by my batch is taking too much time to run. Earlier when i fire the same query it took around 5 minutes and now it is taking around 22 minutes. I cant change the query. Please suggest the correct way to investigate this issue or kindly suggest any resolution.
There is a chance that your job might be waiting for resources to be released by other jobs running in the cluster. Its worth checking in RM UI once you execute the query until the state changes to "RUNNING" - where most of the time is being spent.
Hi Yukti Agrawal,
Based on the logs that you have attached, it seems that,
1. HA is enabled in your cluster
2. One of your NameNode is down
Reason : Hive is not able to reach the active NameNode or None of your NameNodes are active at the time of the process.
1. If NN automatic failover is not enabled, try to manually make the standby NN as active.
2. If NN automatic failover is enabled, try to make Hive aware that NN HA is enabled. The NameNode's IP should be replaced by NameNode's Namespace (HA way of reaching NameNode).
Log Snippet :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
Couple of things you can check
@Rajesh Balamohan The thing is nothing is changed in code or in the volume of data. There is no change made on cluster in terms of configuration or installation of new things. And since there is no change made than how suddenly the job is getting slow?
@Rajesh Balamohan, I am also facing slowness as days progresses, initially the query was taking 15 mins to complete, which now takes 50 mins to complete. Unlike the problem stated here, my dataset is also growing in size. I am attaching the dumps of formatted explain. Could you please look into it and suggest if anything suspicious found. TIA.
@DE_2021, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.