Howto: Adjust hadoop client logging level without restarting components

[#1]

It's sometimes useful to adjust the log level when executing commands from the client to get more info about what is happening. This article explains how that can be achieved:

You can set DEBUG level for any client command by exporting the following environment variable before it is executed:

HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console

Now try executing a client command and watch the stream of DEBUG info come to your terminal. For example:

# hdfs dfs -ls

16/12/24 15:01:34 DEBUG util.Shell: setsid exited with exit code 0
16/12/24 15:01:34 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.5.3.0-37/hadoop/hadoop-common-2.7.3.2.5.3.0-37.jar!/core-default.xml
16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@149494d8
16/12/24 15:01:34 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.5.3.0-37/0/core-site.xml
16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@28c4711c
16/12/24 15:01:35 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
16/12/24 15:01:35 DEBUG security.Groups:  Creating new Groups object
16/12/24 15:01:35 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/12/24 15:01:35 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
16/12/24 15:01:35 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
16/12/24 15:01:35 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
16/12/24 15:01:35 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
16/12/24 15:01:35 DEBUG security.UserGroupInformation: hadoop login
16/12/24 15:01:35 DEBUG security.UserGroupInformation: hadoop login commit
16/12/24 15:01:35 DEBUG security.UserGroupInformation: using kerberos user:[email protected]
16/12/24 15:01:35 DEBUG security.UserGroupInformation: Using user: "[email protected]" with name [email protected]
16/12/24 15:01:35 DEBUG security.UserGroupInformation: User entry: "[email protected]"
16/12/24 15:01:35 DEBUG security.UserGroupInformation: UGI loginUser:[email protected] (auth:KERBEROS)
<snip> 

Note, the following log levels are supported: ALL DEBUG ERROR FATAL INFO OFF TRACE TRACE_INT WARN