Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Regarding"http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/"

avatar
Contributor

Right now HDFS services are getting failed please see the logs-

0.1024069 secs] 91101K->9945K(245760K), 0.1024744 secs] [Times: user=0.32 sys=0.00, real=0.10 secs] 
Heap
 par new generation   total 92160K, used 35183K [0x00000000f0600000, 0x00000000f6a00000, 0x00000000f6a00000)
  eden space 81920K,  34% used [0x00000000f0600000, 0x00000000f218e640, 0x00000000f5600000)
  from space 10240K,  68% used [0x00000000f5600000, 0x00000000f5ccd8e8, 0x00000000f6000000)
  to   space 10240K,   0% used [0x00000000f6000000, 0x00000000f6000000, 0x00000000f6a00000)
 concurrent mark-sweep generation total 153600K, used 2978K [0x00000000f6a00000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21376K, capacity 21678K, committed 21884K, reserved 1069056K
  class space    used 2556K, capacity 2654K, committed 2688K, reserved 1048576K
==> /var/log/hadoop/hdfs/gc.log-201611240749 <==
Java HotSpot(TM) 64-Bit Server VM (25.77-b03) for linux-amd64 JRE (1.8.0_77-b03), built on Mar 20 2016 22:00:46 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 9064548k(2520228k free), swap 5119996k(5119996k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:InitialHeapSize=262144000 -XX:MaxHeapSize=262144000 -XX:MaxNewSize=104857600 -XX:MaxTenuringThreshold=6 -XX:NewSize=52428800 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -XX:ParallelGCThreads=8 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2016-11-24T07:49:27.202+0000: 1.402: [GC (Allocation Failure) 2016-11-24T07:49:27.203+0000: 1.402: [ParNew: 81920K->9191K(92160K), 0.0138715 secs] 81920K->9191K(245760K), 0.0139731 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
2016-11-24T07:49:27.694+0000: 1.893: [GC (Allocation Failure) 2016-11-24T07:49:27.694+0000: 1.893: [ParNew: 91111K->7299K(92160K), 0.0262494 secs] 91111K->10275K(245760K), 0.0263202 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
Heap
 par new generation   total 92160K, used 35518K [0x00000000f0600000, 0x00000000f6a00000, 0x00000000f6a00000)
  eden space 81920K,  34% used [0x00000000f0600000, 0x00000000f218ecd8, 0x00000000f5600000)
  from space 10240K,  71% used [0x00000000f5600000, 0x00000000f5d20d30, 0x00000000f6000000)
  to   space 10240K,   0% used [0x00000000f6000000, 0x00000000f6000000, 0x00000000f6a00000)
 concurrent mark-sweep generation total 153600K, used 2976K [0x00000000f6a00000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 21335K, capacity 21622K, committed 21884K, reserved 1069056K
  class space    used 2555K, capacity 2654K, committed 2688K, reserved 1048576K
1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Master Mentor

@Bibhas Burman

You seem to have run out of disk space ! CAn you run this command and check the output assuming you are root

# su - hdfs
$ hdfs dfsadmin -report

or

hdfs dfs -du -h /

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login