Support Questions
Find answers, ask questions, and share your expertise

What are hive's operation logs used for?

Highlighted

What are hive's operation logs used for?

Rising Star

In hive, there is the option to enable or disable operation logging: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-HiveServer2L...

What are these logs for?

Some background on why I'm asking:

We recently ran into an issue where hiveserver2 was crashing due to huge number of open operation log files (same issue as https://community.hortonworks.com/questions/48351/hiveserver2-hive-users-nofile-ulimit-above-64000.h...). The files all appear to be empty AND they are just being kept open indefinitely. We are going to disable the logs by setting hive.server2.logging.operation.enabled to false but want to know the impact of doing so.

3 REPLIES 3
Highlighted

Re: What are hive's operation logs used for?

@Kit Menke From what I can surmise the Hiveserver2 logging only applies to beeline sessions. Also, when you enable logging you would also need to enable the logging level in order to see any log entries https://issues.apache.org/jira/browse/HIVE-10119. Problem is I can't find those settings in either HDP 2.4 or 2.5.

Highlighted

Re: What are hive's operation logs used for?

Rising Star

I've found two places so far where the operation logs are being used:

  • Ambari Hive View -> Logs tab normally displays output like the number of mappers/reducers.
    • After disabling operation logging the log tab displays nothing.
  • Beeline connections -> After executing a query normally some output is displayed.
    • After disabling operation logging there is no output except the results of the query.

Would be nice to get these back without compromising hiveserver2!

Highlighted

Re: What are hive's operation logs used for?

New Contributor

hello, anyone knows the root cause of why there are so many open operation log files ? Any way to fix that ?