Re: Understanding HBase HDFS usage

tyu · ‎03-11-2018

Compactor calls getCompactionCompressionType() and uses it during compaction: this.compactionCompression = (this.store.getColumnFamilyDescriptor() == null) ? Compression.Algorithm.NONE : this.store.getColumnFamilyDescriptor().getCompactionCompressionType(); While getCompressionType() is used by the flusher - see code in DefaultStoreFlusher#flushSnapshot(): writer = store.createWriterInTmp(cellsCount, store.getColumnFamilyDescriptor().getCompressionType(), false, true, snapshot.isTagsPresent(), false);

tyu · ‎03-11-2018

Please read the following: https://blogs.apache.org/hbase/entry/the_effect_of_columnfamily_rowkey http://hadoop-hbase.blogspot.com/2016/02/hbase-compression-vs-blockencoding_17.html

tyu · ‎10-27-2017

Did you mean that the application spends more time in computation compared to retrieval of information stored in hbase ? Can you describe your use case in more detail ?

tyu · ‎10-27-2017

hbase.backup.enabled should be set to true in hbase-site.xml which is deployed on the cluster.

tyu · ‎10-23-2017

Please take a look at: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_spark-component-guide/content/spark-on-hbase.html

tyu · ‎10-15-2017

Which version of HDP are you using ? Can you post the full stack trace ?

tyu · ‎10-11-2017

Is hbase.replication.conf.dir defined on every node in the peer cluster ? Specifying /tmp is not good idea since the files under /tmp would be periodically cleaned.

tyu · ‎10-06-2017

You should also consider the space occupied by archived files. e.g. /apps/hbase/data/archive/data/default/50m where 50m is table name and /apps/hbase/data is the rootdir

tyu · ‎10-02-2017

kafka-consumer-groups.sh calls ConsumerGroupCommand Please take a look at the following method: ConsumerGroupCommand#getPartitionOffsets

tyu · ‎10-02-2017

Which release of HDP did you install ? Can you attach relevant portion of hbase master log here ? Thanks

tyu · ‎10-02-2017

Ambari UI should display the status for hbase servers

tyu · ‎09-19-2017

From hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java : public static final String NAME = "completebulkload"; This class is in hbase-server jar

tyu · ‎09-19-2017

Please take a look at http://hbase.apache.org/book.html#arch.bulk.load

tyu · ‎09-19-2017

hfile can be loaded thru bulk load.

tyu · ‎09-17-2017

Please take a look at past thread: http://search-hadoop.com/m/HBase/YGbbj6sIj2Ecwi62?subj=Re+ERROR+Can+t+get+master+address+from+ZooKeeper+znode+data+null

tyu · ‎09-16-2017

Cassandra doesn't use hdfs Comparison between Cassandra and hbase is sometimes subjective. e.g. https://stackoverflow.com/questions/14950728/why-hbase-is-a-better-choice-than-cassandra-with-hadoop Please let us know your use case so that better advice can be given. Note: hbase has Replica Read feature which makes hbase highly available.

tyu · ‎09-14-2017

HFileOutputFormat2 is in hbase-server in hbase 1.1.x The mvn dependency:tree output for hbase-server is 200 lines long. You can get the output yourself (for your project). If you encounter problem, let me know.

tyu · ‎09-14-2017

Can you take a look at hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java getNewWriter() shows you how to create StoreFile.Writer

tyu · ‎09-14-2017

Can you take a look at: http://hbase.apache.org/book.html#_writing_hfiles_directly_during_bulk_import which points to: http://hbase.apache.org/book.html#arch.bulk.load

tyu · ‎09-13-2017

bq. Thread.sleep("sometime"); How long was the sleep ? I might be possible that compaction finished after the sleep. Which HDP version are you using ?

tyu · ‎09-13-2017

Before PHOENIX-3288 is resolved, please set phoenix.schema.isNamespaceMappingEnabled to true. hbase-site.xml should be on the classpath

tyu · ‎09-12-2017

For hbase, you can obtain configuration from the following URL: master-node:16010/conf

tyu · ‎09-11-2017

bq. closing the JDBC connection properly in finally blocks Mind sharing the relevant code ? Thanks

tyu · ‎09-11-2017

bq. increase in the number of open files in for the Java process. Which process did you refer to ? I assume you were talking about your application.

tyu · ‎01-04-2017

bq. is there a different version for Netty required ? As I said above, HDP 2.4 came with 3.2.4.Final which is different from 3.9.4.Final (opentsdb built with)

tyu · ‎01-03-2017

bq. OpenTSDB only has Netty 3.2.4 jar But the list below showed netty-3.9.4.Final.jar

tyu · ‎01-03-2017

As you can see, there are two netty jars on the classpath with different version. I am not aware of plan to support OpenTSDB. See if you can build OpenTSDB with the netty version hbase uses. Shading is another option but I am not expert there.

tyu · ‎01-03-2017

Here is HDP 2.4's dependency: [INFO] org.apache.hbase:hbase-it:jar:1.1.2 ... [INFO] +- org.jboss.netty:netty:jar:3.2.4.Final:compile

tyu · ‎01-03-2017

The above seems to indicate version mismatch of netty between OpenTSDB and HDP 2.4 Can you get the version of netty for OpenTSDB 2.2.1 ? Thanks

tyu · ‎01-03-2017

bq. HBase versions shipped with HDP2.4 and 2.5, both say 1.1.2 Both 2.4 and 2.5 were based on hbase 1.1.2 but 2.5 had a lot more fixes / backports (MOB being one of them)

Online	Offline
Last Visited	‎09-14-2018 10:46 PM

Date Registered	‎10-22-2015 03:57 PM
Last Visited	‎09-14-2018 10:46 PM
Total Messages Posted	241
Total Kudos Received	85

Re: Understanding HBase HDFS usage

Re: Hbase MOB and HDP2.4

Re: HBase jars with conflicting versions?

Re: HBase client failing to connect to Kerberized ...

Re: Can we create HBase table using PIG if Yes the...

Re: Difference between CompactionCompression and c...

Re: Understanding HBase HDFS usage

Re: How HBase Region Servers hardware configuratio...

Re: Hbase backup to s3 not working

Re: HDP Spark HBase Connector

Re: Recreate SYSTEM tables Hbase

Re: Help setting up HBase Bulkload Replication

Re: HBASE Performance Tuning Compression

Re: Getting kafka consumer offsets of a consumer g...

Re: HBase servers "disconnected" after loss of Zoo...

Re: Hbase query in HDP 2.3

Re: how to append row key in Hbase using KeyValue ...

Re: how to append row key in Hbase using KeyValue ...

Re: how to append row key in Hbase using KeyValue ...

Re: Im getting Error: can't get master address fro...

Re: How to install Cassandra on HDP 2.6?

Re: how to append row key in Hbase using KeyValue ...

Re: how to append row key in Hbase using KeyValue ...

Re: how to append row key in Hbase using KeyValue ...

Re: Is there a way to find the completion of major...

Re: hive and hbase issue.

Re: How to download HBase configuration files prog...

Re: Phoenix errors on Hbase slow down not closing ...

Re: Phoenix errors on Hbase slow down not closing ...

Re: OpenTSDB startup error -> Caused by: java.lang...

Re: OpenTSDB startup error -> Caused by: java.lang...

Re: OpenTSDB startup error -> Caused by: java.lang...

Re: OpenTSDB startup error -> Caused by: java.lang...

Re: OpenTSDB startup error -> Caused by: java.lang...

Re: Hbase MOB and HDP2.4