Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase performance issue

Hbase performance issue

New Contributor

Hi All,

 

I have setup of hadoop cluster(CDH5.4.4) with hbase and phoenix. I see these warnings in my regionserver logs:

 

2016-06-27 10:28:51,987 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog: Slow sync cost: 477 ms, current pipeline: [DatanodeInfoWithStorage[XXX.xx.xx.xxx:50010,DS-bf645817-6164-4723-952a-161a65da092d,DISK], DatanodeInfoWithStorage[XXX.XX.xx.xxx:50010,DS-dd3f32e1-9b62-4f95-b1e2-5fd1a1c2a6e2,DISK]]

I have analyzed the logs slow sync cost goes upto 2sec.

I also see these warnings:

2016-06-27 10:28:51,987 WARN org.apache.hadoop.hbase.ipc.RpcServer: (responseTooSlow): {"processingtimems":540,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"XXX.XXX.XXX.XXX:49410","starttimems":1467003531447,"queuetimems":0,"class":"HRegionServer","responsesize":14,"method":"Multi"}  

This call is coming from client for PUT request.

 

I see these warnings in my datanode logs:

2016-06-27 10:47:51,162 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync took 380ms (threshold=300ms), isSync:false, flushTotalNanos=379910826ns

2016-06-27 10:47:51,162 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:400ms (threshold=300ms)

2016-06-27 10:47:51,162 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:397ms (threshold=300ms)

 

 

 

  • My cluster configuration is: 4 datanodes, 2 namenodes. 
    • Datanode Configuration: 64gb RAM, NON-RAID Disks 4.5TB each, 32 cores 
    • Namenode Configuration: 32gb RAM, 32 cores 
    • Regionservers co-hosted with datanode
    • HBase master cohosted with namenode
    • Hbase regionserver XMX-24 gb
  • I would appreciate any pointers for resolving this issue.
2 REPLIES 2
Highlighted

Re: Hbase performance issue

Expert Contributor

Is there something else consuming disk time on the datanodes? (YARN/MR seems like a likely candidate)

Is it consistently one datanode?

Are you disabling periodic compactions and scheduling them instead?

are your datanode java memory settings enough? (lots of garbage collection?)

How many regions per regionserver are you running?

 

Re: Hbase performance issue

New Contributor

YARN/MR are not running.

No, its coming on all data nodes.

Yes, I have turned major compaction to 0.

I have given 24 gb ram to hbase. I am using g1gc. These are my g1gc parameters.

 

-Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1024m -ea -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc-hbase.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=10102 -XX:MaxGCPauseMillis=600 -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=10 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -Xms24g -Xmx24g

 

10 regions per regionserver.

 

My IPC time: http://imgur.com/iwKnFNe This is the main issue I am facing.