Support Questions

kabalax · ‎09-17-2018

Hi,

HDP-2.6.1.0 cluster running on CentOS 7.5.1804.

I get the following log file growing under /tmp on the Knox server. The full path is /tmp/username/hive.log (where username is the one accessing Hive).

The sample content of hive.log is provided below. Although the log file is rotated I'd like to move it to /var/log, so I was wondering which section of Hive(?) configuration is responsible for that log file? I see multiple traces of hive.log.dir and hive.log.file but I'm not sure which one is relevant.

Many thanks.

2018-09-12 13:07:08,943 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=TezRunVertex.Reducer 2 start=1536750381000 end=1536750428943 duration=47943 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:08,944 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=TezRunVertex.Reducer 5 start=1536750368334 end=1536750428944 duration=60610 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:08,944 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=TezRunVertex.Reducer 6 start=1536750419768 end=1536750428944 duration=9176 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:08,944 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=TezRunVertex.Reducer 7 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:08,944 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=TezRunVertex.Reducer 7 start=1536750428944 end=1536750428944 duration=0 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:09,008 INFO  [main]: monitoring.TezJobMonitor$UpdateFunction (TezJobMonitor.java:update(137)) - Map 1: 10/10   Map 3: 29/29    Map 4: 10/10    Map 8: 29/29    Reducer 2: 1/1  Reducer 5: 1/1  Reducer 6: 1/1  Reducer 7: 1/1
2018-09-12 13:07:09,009 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=TezRunDag start=1536750115791 end=1536750429009 duration=313218 from=org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor>
2018-09-12 13:07:09,106 INFO  [main]: counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000
2018-09-12 13:07:09,186 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=RemoveTempOrDuplicateFiles from=FileSinkOperator>
2018-09-12 13:07:09,193 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=RemoveTempOrDuplicateFiles start=1536750429186 end=1536750429193 duration=7 from=FileSinkOperator>
2018-09-12 13:07:09,194 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=RenameOrMoveFiles from=FileSinkOperator>
2018-09-12 13:07:09,194 INFO  [main]: exec.FileSinkOperator (Utilities.java:mvFileToFinalPath(2026)) - Moving tmp dir: hdfs://HADOOP/tmp/hive/svc-feed/ac4bf6dd-f7be-4ded-b972-01ea88f8fe6b/hive_2018-09-12_13-01-35_168_1425040625106206929-1/-mr-10001/.hive-staging_hive_2018-09-12_13-01-35_168_1425040625106206929-1/_tmp.-ext-10002 to: hdfs://HADOOP/tmp/hive/svc-feed/ac4bf6dd-f7be-4ded-b972-01ea88f8fe6b/hive_2018-09-12_13-01-35_168_1425040625106206929-1/-mr-10001/.hive-staging_hive_2018-09-12_13-01-35_168_1425040625106206929-1/-ext-10002
2018-09-12 13:07:09,236 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=RenameOrMoveFiles start=1536750429194 end=1536750429236 duration=42 from=FileSinkOperator>
2018-09-12 13:07:09,265 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=runTasks start=1536750107335 end=1536750429265 duration=321930 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,266 INFO  [main]: hooks.ATSHook (ATSHook.java:<init>(114)) - Created ATS Hook
2018-09-12 13:07:09,266 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,268 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1536750429266 end=1536750429268 duration=2 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,268 INFO  [main]: ql.Driver (Driver.java:execute(1638)) - Resetting the caller context to 
2018-09-12 13:07:09,269 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=Driver.execute start=1536750106856 end=1536750429269 duration=322413 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,270 INFO  [main]: ql.Driver (SessionState.java:printInfo(984)) - OK
2018-09-12 13:07:09,270 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,270 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=releaseLocks start=1536750429270 end=1536750429270 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,270 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=Driver.run start=1536750095097 end=1536750429270 duration=334173 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,272 INFO  [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(362)) - Received post-hook notification for :svc-feed_20180912130135_9b82b03b-0581-4976-88a6-b2a58ff9251a
2018-09-12 13:07:09,302 INFO  [main]: exec.ListSinkOperator (Operator.java:close(616)) - Closing operator OP[60]
2018-09-12 13:07:09,368 INFO  [main]: CliDriver (SessionState.java:printInfo(984)) - Time taken: 334.184 seconds
2018-09-12 13:07:09,369 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,369 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=releaseLocks start=1536750429369 end=1536750429369 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2018-09-12 13:07:09,382 INFO  [main]: tez.TezSessionPoolManager (TezSessionPoolManager.java:close(183)) - Closing tez session default? false
2018-09-12 13:07:09,382 INFO  [main]: tez.TezSessionState (TezSessionState.java:close(293)) - Closing Tez Session
2018-09-12 13:07:09,383 INFO  [main]: client.TezClient (TezClient.java:stop(518)) - Shutting down Tez Session, sessionName=HIVE-ac4bf6dd-f7be-4ded-b972-01ea88f8fe6b, applicationId=application_1503845958062_15700

ssubhas · ‎09-18-2018

@Teddy Brewski

Below are the properties which control the logs and other files written to /tmp/<username> folder.

<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
</property>

<property>
<name>hive.exec.local.scratchdir</name>
<value>/var/log/hadoop/hive/tmp/${user.name}</value>
</property>

<property>
<name>hive.downloaded.resources.dir</name>
<value>/var/log/hadoop/hive/tmp/hive/${hive.session.id}_resources</value>
</property>

<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/var/log/hadoop/hive/tmp/operations_logs</value>
</property>

You can add/modify under Ambari -> Hive configs.

Cloudera Community

Support Questions

hive.log under /tmp