Support Questions

Find answers, ask questions, and share your expertise

How disable [main] DEBUG org.apache.hadoop in CMD Hive?

avatar
Explorer

I did add AUX Jar to Hive in section Advanced hive-env parameter hive-env template in end listing

export HIVE_AUX_JARS_PATH="$HIVE_AUX_JARS_PATH,/opt/libs/spark-1.1.7.jar"

After this actions in console Hive did appear many DEBUG information...

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/libs/spark-1.1.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.1.0-129/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
20:09:06.519 [main] DEBUG org.apache.hadoop.util.Shell - setsid exited with exit code 0
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.1.0-129/0/hive-log4j.properties
20:09:07.637 [main] DEBUG org.apache.hadoop.security.authentication.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty
20:09:07.675 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login
20:09:07.675 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit
20:09:07.680 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: user1
20:09:07.680 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: user1" with name user1
20:09:07.680 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - User entry: "user1"
20:09:07.680 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Assuming keytab is managed externally since logged in from subject.
20:09:07.681 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:user1 (auth:SIMPLE)
20:09:08.466 [Finalizer] DEBUG org.apache.hadoop.fs.azure.NativeAzureFileSystem - finalize() called.
20:09:08.467 [Finalizer] DEBUG org.apache.hadoop.fs.azure.NativeAzureFileSystem - finalize() called.
20:09:08.763 [main] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
20:09:09.253 [Thread-7] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL client skipping handshake in unsecured configuration for addr = /192.168.63.22, datanodeId = DatanodeInfoWithStorage[192.168.63.22:50010,DS-13731d20-37a7-4e96-adc7-22883dc36760,DISK]
20:09:09.953 [Thread-10] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL client skipping handshake in unsecured configuration for addr = /192.168.63.22, datanodeId = DatanodeInfoWithStorage[192.168.63.22:50010,DS-13731d20-37a7-4e96-adc7-22883dc36760,DISK]
20:09:11.015 [main] INFO org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, version=0.7.0.2.6.1.0-129, revision=bbcfb9e8d9cc93fb586b32199eb9492528449f7c, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2017-05-31T03:06:59Z ]
20:09:11.111 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:user1 (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:163)
20:09:11.309 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:user1 (auth:SIMPLE) from:org.apache.hadoop.yarn.client.AHSProxy.getProxy(AHSProxy.java:49)
20:09:11.319 [main] INFO org.apache.tez.client.TezClient - Session mode. Starting session.
20:09:11.323 [main] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: /hdp/apps/2.6.1.0-129/tez/tez.tar.gz
20:09:11.506 [main] INFO org.apache.tez.client.TezClient - Tez system stage directory hdfs://server01:8020/tmp/hive/user1/_tez_session_dir/864e3b98-3cfd-4a8e-a487-51668116d7e0/.tez/application_1526574235371_0024 doesn't exist and is created
20:09:11.509 [main] DEBUG org.apache.tez.client.TezClientUtils - AppMaster capability = <memory:512, vCores:1>
20:09:11.510 [main] DEBUG org.apache.tez.client.TezClientUtils - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java  -Xmx409m -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-129 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
20:09:11.555 [Thread-13] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL client skipping handshake in unsecured configuration for addr = /192.168.63.22, datanodeId = DatanodeInfoWithStorage[192.168.63.22:50010,DS-13731d20-37a7-4e96-adc7-22883dc36760,DISK]
20:09:11.667 [Thread-15] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL client skipping handshake in unsecured configuration for addr = /192.168.63.22, datanodeId = DatanodeInfoWithStorage[192.168.63.22:50010,DS-13731d20-37a7-4e96-adc7-22883dc36760,DISK]
20:09:13.186 [main] INFO org.apache.tez.client.TezClient - The url to track the Tez Session: http://server01:8088/proxy/application_1526574235371_0024/

How disable this DEBUG information in hive?

1 REPLY 1

avatar
Contributor

Log levels are confgured in log4j files located in /etc/hive/conf/ and /etc/hive/conf/conf.server

Go hiveserver2 node and run

cd /etc/hive/conf/ 
grep -R DEBUG * | grep hive.root.logger

you will see a file called hive-log4j.properties

Edit that file and change the line called hive.root.logger from DEBUG to INFO