Member since
07-05-2017
13
Posts
0
Kudos Received
0
Solutions
11-15-2017
05:47 AM
Thanks for the reply. But i need to know the basic proposition atleast. Say I'm having 20 GB data in my hadoop and quite few jobs. So now what might be the good RAM size for all the components which i have metioned .
... View more
11-14-2017
05:25 PM
Could someone clarify how to select the RAM size for below Hadoop components? HADOOP: Namenode,Datanode HBase: HbaseMaster, RegionServer Zookeeper
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
09-16-2017
02:13 PM
To ensure major compaction is completed: Is hbase shell major_compact 'tableName' is synchronous call?. Or this command is just triggering the compaction process and it may take some time to complete the compaction.
... View more
- Tags:
- Hadoop Core
- HBase
Labels:
- Labels:
-
Apache HBase
09-14-2017
01:20 PM
Hadoop: 2.7.3 , HBase 1.3.1 Major compaction has been triggered for all the tables in hbase via hbase client. for(String table: listofTables) admin.majorCompact(table) Compaction has been triggered for all the tables which can be confirmed from hbase logs, but completion logs are not present for all the tables(checked logs even after two hours of compaction trigger)
... View more
09-13-2017
12:50 PM
We have read the Hfile, and it has all the values(key,cf,cq,value).Why the Hfile still retains the data though, all the data has been deleted and even the count of table is zero? Could you explain how to look for delete markers?
... View more
09-13-2017
11:13 AM
1) recovered.edits is not holding up the space, Clearly HFile is having some data(deleted row keys,columns,column qualifier) 0 /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4/recovered.edits 2) KEEP_DELETED_CELLS is 'false' 3) executed flush command too. Still the issue persists.
... View more
09-13-2017
08:26 AM
After polling for some time, if the state returns MAJOR, shall we confirm major compaction is completed?. admin.majorCompact(TableName.valueOf("testTable")); Thread.sleep("sometime"); CompactionState state=admin.getCompactionState("testTable"); if(state.equals(CompactionState.MAJOR)) { //Major compaction is completed } This code always returns state as NONE
... View more
09-13-2017
07:57 AM
-The size of HFiles for an table with 3 lakh enteries is 1.2G, $ hdfs dfs -du -h /hbase/data/default/testTable/ 806 /hbase/data/default/testTable/.tabledesc
0 /hbase/data/default/testTable/.tmp 1.2 M /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4 1.2 G /hbase/data/default/testTable/21c5017b57212f76672080e8e3f0724e - Deleted all the entries in the table and triggered major compaction via hbase client. After this, size of Hfiles are, $ hdfs dfs -du -h /hbase/data/default/testTable/ 806 /hbase/data/default/testTable/.tabledesc 0 /hbase/data/default/testTable/.tmp 127.3 M /hbase/data/default/testTable/0bce2f3457622bf79f75222e9c3107a4 217.4 M /hbase/data/default/testTable/21c5017b57212f76672080e8e3f0724e We could see some of the entries in the HFile that are deleted, though the count of the table is zero. What could be the cause for this scenario.(HFile holding few entries even after deletion and compaction with the size of around 300Mb.)
... View more
Labels:
- Labels:
-
Apache HBase
09-13-2017
07:12 AM
Hey Ankit, thanks for your response. Hbase Client returns NONE though the compaction is completed for the table(which can be identifed from the logs). And major compaction is an asychronous process, how to predict the wait time to get the state form the client API (In case if it returns the state properly)
... View more
09-13-2017
06:03 AM
I want to trigger major compaction for all the tables in the hbase. Using hbase client API, major compaction is triggered for all the times via admin.majorCompact(tableName). 1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process.2) Is it mandatory to wait until compaction process completion , to query hbase for real time process
... View more
Labels:
- Labels:
-
Apache HBase
07-06-2017
10:26 PM
After restarting, again region server went down. This time it's not because of the JVM pause. Is it again because of the kernel, which blocks on a flush of WAL's (after random delay 238839ms)?? 2017-07-05 16:57:11,117 INFO [server.com,16201,1499221788437_ChoreService_1] regionserver.HRegionServer: server.com,16201,1499221788437-MemstoreFlusherChore requesting flush of __test_table_server.com__,,1496366805684.7d0dbb83f455b1bda1a0b9efa4e9719c. because __status_family___ has an old edit so flush to free WALs after random delay 238839ms
2017-07-05 16:58:24,389 INFO [regionserver/server.com/1.1.1.1:16201-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 80029ms for sessionid 0x15c6668db396db5, closing socket connection and attempting reconnect
2017-07-05 16:58:24,390 INFO [server.com,16201,1499221788437_ChoreService_1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-07-05 16:58:24,390 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 80008ms for sessionid 0x15c6668db396db4, closing socket connection and attempting reconnect
2017-07-05 16:58:24,412 INFO [server.com,16201,1499221788437_ChoreService_1] hbase.ScheduledChore: Chore: server.com,16201,1499221788437-MemstoreFlusherChore missed its start time
2017-07-05 16:58:24,415 INFO [server.com,16201,1499221788437_ChoreService_4] regionserver.HRegionServer: server.com,16201,1499221788437-MemstoreFlusherChore requesting flush of __test_table_server.com__,,1496366805684.7d0dbb83f455b1bda1a0b9efa4e9719c. because __status_family___ has an old edit so flush to free WALs after random delay 114843ms
2017-07-05 16:58:24,415 INFO [server.com,16201,1499221788437_ChoreService_3] hbase.ScheduledChore: Chore: server.com,16201,1499221788437-HeapMemoryTunerChore missed its start time
2017-07-05 16:58:26,224 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-07-05 16:58:26,225 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-07-05 16:58:26,227 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c6668db396db4 has expired, closing socket connection
... View more
07-05-2017
06:48 PM
favorite
Region server logs WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 129793ms No GCs detected Since it's clear that, no GCs has been detected. What could be the reason for the JVM pause.How to find out the reason for the same? Additional Logs:
2017-07-02 05:17:19,754 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143154ms for sessionid 0x15c6668db38d116, closing socket connection and attempting reconnect
2017-07-02 05:17:19,754 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143154ms for sessionid 0x15c6668db38d116, closing socket connection and attempting reconnect
2017-07-02 05:17:19,760 WARN [ResponseProcessor for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755625_26457] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755625_26457
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
2017-07-02 05:17:19,756 INFO [regionserver/server.com/1.1.1.1:16201-SendThread(localhost:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 143363ms for sessionid 0x15c6668db38d11b, closing socket connection and attempting reconnect
2017-07-02 05:17:19,769 INFO [server.com,16201,1498621362582_ChoreService_2] hbase.ScheduledChore: Chore: server.com,16201,1498621362582-MemstoreFlusherChore missed its start time
2017-07-02 05:17:19,769 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 129793ms
No GCs detected
2017-07-02 05:17:19,770 INFO [server.com,16201,1498621362582_ChoreService_3] hbase.ScheduledChore: Chore: server.com,16201,1498621362582-HeapMemoryTunerChore missed its start time
2017-07-02 05:17:19,769 INFO [server.com,16201,1498621362582_ChoreService_1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-07-02 05:17:19,760 WARN [ResponseProcessor for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755624_26456] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-2076318769-1.1.1.1-1496366779987:blk_1073755624_26456
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798)
... View more
Labels:
- Labels:
-
Apache HBase