Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

What are hive's operation logs used for?

avatar
Expert Contributor

In hive, there is the option to enable or disable operation logging: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-HiveServer2L...

What are these logs for?

Some background on why I'm asking:

We recently ran into an issue where hiveserver2 was crashing due to huge number of open operation log files (same issue as https://community.hortonworks.com/questions/48351/hiveserver2-hive-users-nofile-ulimit-above-64000.h...). The files all appear to be empty AND they are just being kept open indefinitely. We are going to disable the logs by setting hive.server2.logging.operation.enabled to false but want to know the impact of doing so.

3 REPLIES 3

avatar

@Kit Menke From what I can surmise the Hiveserver2 logging only applies to beeline sessions. Also, when you enable logging you would also need to enable the logging level in order to see any log entries https://issues.apache.org/jira/browse/HIVE-10119. Problem is I can't find those settings in either HDP 2.4 or 2.5.

avatar
Expert Contributor

I've found two places so far where the operation logs are being used:

  • Ambari Hive View -> Logs tab normally displays output like the number of mappers/reducers.
    • After disabling operation logging the log tab displays nothing.
  • Beeline connections -> After executing a query normally some output is displayed.
    • After disabling operation logging there is no output except the results of the query.

Would be nice to get these back without compromising hiveserver2!

avatar
Explorer

hello, anyone knows the root cause of why there are so many open operation log files ? Any way to fix that ?