Re: How to select proper RAM size for hadoop comp...

vishnu_p · ‎11-15-2017

Thanks for the reply. But i need to know the basic proposition atleast. Say I'm having 20 GB data in my hadoop and quite few jobs. So now what might be the good RAM size for all the components which i have metioned .

vishnu_p · ‎11-14-2017

Could someone clarify how to select the RAM size for below Hadoop components? HADOOP: Namenode,Datanode HBase: HbaseMaster, RegionServer Zookeeper

vishnu_p · ‎09-16-2017

To ensure major compaction is completed: Is hbase shell major_compact 'tableName' is synchronous call?. Or this command is just triggering the compaction process and it may take some time to complete the compaction.

vishnu_p · ‎09-14-2017

Hadoop: 2.7.3 , HBase 1.3.1 Major compaction has been triggered for all the tables in hbase via hbase client. for(String table: listofTables) admin.majorCompact(table) Compaction has been triggered for all the tables which can be confirmed from hbase logs, but completion logs are not present for all the tables(checked logs even after two hours of compaction trigger)

vishnu_p · ‎09-13-2017

We have read the Hfile, and it has all the values(key,cf,cq,value).Why the Hfile still retains the data though, all the data has been deleted and even the count of table is zero? Could you explain how to look for delete markers?

vishnu_p · ‎09-13-2017

1) recovered.edits is not holding up the space, Clearly HFile is having some data(deleted row keys,columns,column qualifier) 0 /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4/recovered.edits 2) KEEP_DELETED_CELLS is 'false' 3) executed flush command too. Still the issue persists.

vishnu_p · ‎09-13-2017

After polling for some time, if the state returns MAJOR, shall we confirm major compaction is completed?. admin.majorCompact(TableName.valueOf("testTable")); Thread.sleep("sometime"); CompactionState state=admin.getCompactionState("testTable"); if(state.equals(CompactionState.MAJOR)) { //Major compaction is completed } This code always returns state as NONE

vishnu_p · ‎09-13-2017

-The size of HFiles for an table with 3 lakh enteries is 1.2G, $ hdfs dfs -du -h /hbase/data/default/testTable/ 806 /hbase/data/default/testTable/.tabledesc 0 /hbase/data/default/testTable/.tmp 1.2 M /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4 1.2 G /hbase/data/default/testTable/21c5017b57212f76672080e8e3f0724e - Deleted all the entries in the table and triggered major compaction via hbase client. After this, size of Hfiles are, $ hdfs dfs -du -h /hbase/data/default/testTable/ 806 /hbase/data/default/testTable/.tabledesc 0 /hbase/data/default/testTable/.tmp 127.3 M /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4 217.4 M /hbase/data/default/testTable/21c5017b57212f76672080e8e3f0724e We could see some of the entries in the HFile that are deleted, though the count of the table is zero. What could be the cause for this scenario.(HFile holding few entries even after deletion and compaction with the size of around 300Mb.)

vishnu_p · ‎09-13-2017

Hey Ankit, thanks for your response. Hbase Client returns NONE though the compaction is completed for the table(which can be identifed from the logs). And major compaction is an asychronous process, how to predict the wait time to get the state form the client API (In case if it returns the state properly)

vishnu_p · ‎09-13-2017

I want to trigger major compaction for all the tables in the hbase. Using hbase client API, major compaction is triggered for all the times via admin.majorCompact(tableName). 1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process.2) Is it mandatory to wait until compaction process completion , to query hbase for real time process

vishnu_p · ‎07-09-2017

vishnu_p · ‎07-06-2017

After restarting, again region server went down. This time it's not because of the JVM pause. Is it again because of the kernel, which blocks on a flush of WAL's (after random delay 238839ms)?? 2017-07-05 16:57:11,117 INFO [server.com,16201,1499221788437_ChoreService_1] regionserver.HRegionServer: server.com,16201,1499221788437-MemstoreFlusherChore requesting flush of __test_table_server.com__,,1496366805684.7d0dbb83f455b1bda1a0b9efa4e9719c. because __status_family___ has an old edit so flush to free WALs after random delay 238839ms 2017-07-05 16:58:24,389 INFO [regionserver/server.com/1.1.1.1:16201-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 80029ms for sessionid 0x15c6668db396db5, closing socket connection and attempting reconnect 2017-07-05 16:58:24,390 INFO [server.com,16201,1499221788437_ChoreService_1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2017-07-05 16:58:24,390 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 80008ms for sessionid 0x15c6668db396db4, closing socket connection and attempting reconnect 2017-07-05 16:58:24,412 INFO [server.com,16201,1499221788437_ChoreService_1] hbase.ScheduledChore: Chore: server.com,16201,1499221788437-MemstoreFlusherChore missed its start time 2017-07-05 16:58:24,415 INFO [server.com,16201,1499221788437_ChoreService_4] regionserver.HRegionServer: server.com,16201,1499221788437-MemstoreFlusherChore requesting flush of __test_table_server.com__,,1496366805684.7d0dbb83f455b1bda1a0b9efa4e9719c. because __status_family___ has an old edit so flush to free WALs after random delay 114843ms 2017-07-05 16:58:24,415 INFO [server.com,16201,1499221788437_ChoreService_3] hbase.ScheduledChore: Chore: server.com,16201,1499221788437-HeapMemoryTunerChore missed its start time 2017-07-05 16:58:26,224 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-05 16:58:26,225 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 2017-07-05 16:58:26,227 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c6668db396db4 has expired, closing socket connection

vishnu_p · ‎07-05-2017

favorite Region server logs WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 129793ms No GCs detected Since it's clear that, no GCs has been detected. What could be the reason for the JVM pause.How to find out the reason for the same? Additional Logs: 2017-07-02 05:17:19,754 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143154ms for sessionid 0x15c6668db38d116, closing socket connection and attempting reconnect 2017-07-02 05:17:19,754 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143154ms for sessionid 0x15c6668db38d116, closing socket connection and attempting reconnect 2017-07-02 05:17:19,760 WARN [ResponseProcessor for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755625_26457] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755625_26457 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) 2017-07-02 05:17:19,756 INFO [regionserver/server.com/1.1.1.1:16201-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143363ms for sessionid 0x15c6668db38d11b, closing socket connection and attempting reconnect 2017-07-02 05:17:19,769 INFO [server.com,16201,1498621362582_ChoreService_2] hbase.ScheduledChore: Chore: server.com,16201,1498621362582-MemstoreFlusherChore missed its start time 2017-07-02 05:17:19,769 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 129793ms No GCs detected 2017-07-02 05:17:19,770 INFO [server.com,16201,1498621362582_ChoreService_3] hbase.ScheduledChore: Chore: server.com,16201,1498621362582-HeapMemoryTunerChore missed its start time 2017-07-02 05:17:19,769 INFO [server.com,16201,1498621362582_ChoreService_1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2017-07-02 05:17:19,760 WARN [ResponseProcessor for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755624_26456] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755624_26456 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)

Online	Offline
Last Visited	‎11-15-2017 05:47 AM

Date Registered	‎07-05-2017 06:48 PM
Last Visited	‎11-15-2017 05:47 AM
Total Messages Posted	13

Re: How to select proper RAM size for hadoop comp...

How to select proper RAM size for hadoop componen...

Is Hbase shell major_compact command synchronous o...

Re: Is there a way to find the completion of major...

Re: What could be the size of HFiles after deletin...

Re: What could be the size of HFiles after deletin...

Re: Is there a way to find the completion of major...

What could be the size of HFiles after deleting al...

Re: Is there a way to find the completion of major...

Is there a way to find the completion of major com...

Get or scan to use for retrieving the data,when i ...

Re: Hbase region server went down because of a JVM...

Hbase region server went down because of a JVM pau...