Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting Client out of cache takes long time

Getting Client out of cache takes long time

Contributor

Over the weekend, I noticed that the cluster was running really slow. All the jobs that otherwise ran fine took longer to complete. Even a simple 

HDFS lookup (hdfs dfs -ls /) took about 20+ seconds. 

 

When I ran the debug, I noticed that it took a lot of time to get the client out of cache

 

[user@host ~]$ HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -ls /
15/10/07 14:37:05 DEBUG util.Shell: setsid exited with exit code 0
15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-common-2.6.0-cdh5.4.4.jar!/core-default.xml
15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7c594d5e
15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@22157ace
15/10/07 14:37:05 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
.
.
.
15/10/07 14:37:06 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@7505062c
15/10/07 14:37:22 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
.
.
.

^ Please notice the time difference highlighted in bold above. It takes about 16s for it to get the client out of cache. Our CPU utilization is very low

(96% idle) and every thing else seems fine. Also, note that we have made no configuration changes on this cluster whatsoever.

Don't have an account?
Coming from Hortonworks? Activate your account here