Re: Understanding HBase HDFS usage

tyu · ‎07-20-2016

You should spread out the major compaction on individual tables. Refer to the following: HBASE-16147 Add ruby wrapper for getting compaction state If there is not enough time to wait for compaction on one table to finish, at least leave some time between the start of major compactions.

tyu · ‎07-19-2016

Here is sample /jmx output: "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", ... "percentFilesLocal" : 97, "percentFilesLocalSecondaryRegions" : 97,

tyu · ‎07-19-2016

Did you mean locality for HFiles ? You can get such metric through /jmx There isn't command line which would directly give you this metric.

tyu · ‎07-18-2016

Please take a look at: https://issues.apache.org/jira/browse/HBASE-11339 which would reduce I/O amplification incurred by medium objects. This feature is in the upcoming HDP 2.5 release.

tyu · ‎07-16-2016

By default, each component (NodeManager, Datanode, and RegionServer daemon) would have their own predetermined heap memory setting. You need to tune yourself.

tyu · ‎07-14-2016

Can you find encoded region name for regions stuck in FAILED_OPEN state and pastebin related region server log ? It would help us understand what caused the region not to open. BTW which HDP version are you running ?

tyu · ‎07-06-2016

2016-07-06 14:59:14,466 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=m2.domain:2181 sessionTimeout=120000 watcher=hconnection- 0x7bc9e6ab0x0, quorum=m2.domain:2181, baseZNode=/hbase-secure Looks like AMS tried to connect to hbase cluster's znode. AMS should use /ams-hbase-secure as base znode. Can you check your configuration ?

tyu · ‎07-05-2016

Can you outline the type of queries involving probability_score and real_labelVal fields ? The answer to the above would determine whether declaring fields as String is good practice.

tyu · ‎07-05-2016

Looks like HBASE-14963 was integrated into HDP 2.3 on Fri Jan 22 Can you post the commit hash of the HDP you use ? Please post the complete stack trace. Thanks

tyu · ‎07-02-2016

Please install Azure storage SDK for Java (com.microsoft.azure:azure-storage)

tyu · ‎07-01-2016

Can you try setting the following before invoking spark-shell ? export HADOOP_HOME=/usr/hdp/current/hadoop-client You can also set the following in conf/log4j.properties: log4j.logger.org.apache.spark.repl.Main=DEBUG so that you can get more information.

tyu · ‎06-29-2016

You can add debug logs in your coprocessor. Enable DEBUG logging on region servers and check the logs after action is performed in shell.

tyu · ‎06-29-2016

Can you describe the function of the coprocessor ? It should be possible to use shell and observe the side effect your coprocessor makes.

tyu · ‎06-27-2016

You can find a ton of hbase coprocessors under: https://github.com/apache/phoenix

tyu · ‎06-27-2016

Where is the custom jar located ? Can you show related snippet from hbase-site.xml ? Thanks

tyu · ‎06-27-2016

In region server log, you should observe something similar to the following: 2016-06-16 19:44:10,222 INFO [regionserver/hbase5-merge-normalizer-3.openstacklocal/172.22.78.125:16020] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.AccessController was loaded successfully with priority (536870911). If the coprocessor was loaded per table, you can use 'describe' command in shell to verify.

tyu · ‎06-23-2016

The functions `phoenixTableAsDataFrame`, `phoenixTableAsRDD` and `saveToPhoenix` all support optionally specifying a `conf` Hadoop configuration parameter with custom Phoenix client settings, as well as an optional `zkUrl` parameter for the Phoenix connection URL. val configuration = new Configuration() // set zookeeper.znode.parent and hbase.zookeeper.quorum in the conf "TABLE1", Seq("ID", "COL1"), conf = configuration

tyu · ‎06-22-2016

As the last ERROR showed you, the effective hbase-site.xml seems to not be on the classpath. The zookeeper ensemble was redacted. Please make sure the quorum is the one used by hbase.

tyu · ‎06-21-2016

Please take a look at hbase-shell//src/test/ruby/hbase/table_test.rb for sample syntax where x is column family: @test_table._scan_internal(FILTER => "SingleColumnValueFilter('x', 'a', >=, 'binary:\x82', true, true)", COLUMNS => 'x:a, x:b'

tyu · ‎06-21-2016

Hive using MR-over-hbase-snapshots would be a viable solution. Perform a snapshot in hbase, then use hive to directly read from the underlying HFiles.

tyu · ‎06-21-2016

You can use SingleColumnValueFilter on 'column1' Meanwhile, specify all the columns you want to retrieve using COLUMNS.

tyu · ‎06-20-2016

bq. Hbase created the columns based on alphabetical order When you query hbase, you observe alphabetical order because that's what hbase stores internally. Do you observe fewer than 10 rows after importing the sample data ?

tyu · ‎06-20-2016

18414 was the region server process. Was it still running ?

tyu · ‎06-20-2016

When I tried 'hbase' command within shell (HDP 2.5), I got: hbase(main):001:0> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv NoMethodError: undefined method `hbase' for #<Object:0x415d88de>

tyu · ‎06-20-2016

hbase.csv should be placed on hdfs.

tyu · ‎06-20-2016

Disabled replication would still hold on to the WAL files because, it guarantees to not lose data between disable and enable. You can execute remove_peer, which frees up the WAL files eligible for deletion. When you re-add replication peer again, the replication would start from the current status, versus if you re-enable a peer, it will continue from where it left off.

tyu · ‎06-20-2016

There are currently two services which may keep the files in the archive directory. First is a TTL process, which ensures that the WAL files are kept at least for 10 min. This is controlled by hbase.master.logcleaner.ttl configuration property in master. The other one is replication. If you had replication setup before, the replication processes will hang on to the WAL files until they are replicated. Even if you disabled the replication, the files are still referenced. You can look at the logs of master from classes (LogCleaner, TimeToLiveLogCleaner, ReplicationLogCleaner) to see whether any exception was thrown.

tyu · ‎06-18-2016

It is work in progress: https://issues.apache.org/jira/browse/NIFI-991

tyu · ‎06-17-2016

hbase(main):001:0> scan 't2' ROW COLUMN+CELL 5842 column=pressure:in, timestamp=1466184191234, value=240 5842 column=pressure:out, timestamp=1466184191234, value=340 5842 column=temp:in, timestamp=1466184191234, value=50 5842 column=temp:out, timestamp=1466184191234, value=30 5842 column=vibration:, timestamp=1466184191234, value=4 1 row(s) in 0.4840 seconds

tyu · ‎06-17-2016

I created a table with schema similar to yours. create 't2', {NAME => 'pressure'}, {NAME => 'temp'}, {NAME => 'vibration'} I created hbase.csv with the line in your first post. I then executed the following (HDP 2.5.0.0-267 which is 1.1.2 with some patches): hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=',' -Dimporttsv.columns='HBASE_ROW_KEY,temp:in,temp:out,vibration,pressure:in,pressure:out' t2 hbase.csv 2016-06-17 17:23:18,142 INFO [main] mapreduce.Job: Running job: job_1462297012839_0003 2016-06-17 17:23:29,332 INFO [main] mapreduce.Job: Job job_1462297012839_0003 running in uber mode : false 2016-06-17 17:23:29,335 INFO [main] mapreduce.Job: map 0% reduce 0% 2016-06-17 17:23:38,466 INFO [main] mapreduce.Job: map 100% reduce 0% 2016-06-17 17:23:38,478 INFO [main] mapreduce.Job: Job job_1462297012839_0003 completed successfully 2016-06-17 17:23:38,670 INFO [main] mapreduce.Job: Counters: 31

Online	Offline
Last Visited	‎09-14-2018 10:46 PM

Date Registered	‎10-22-2015 03:57 PM
Last Visited	‎09-14-2018 10:46 PM
Total Messages Posted	241
Total Kudos Received	85

Re: Understanding HBase HDFS usage

Re: Hbase MOB and HDP2.4

Re: HBase jars with conflicting versions?

Re: HBase client failing to connect to Kerberized ...

Re: Can we create HBase table using PIG if Yes the...

Re: Run hbase major compaction on all tables

Re: hbase local files via ambari

Re: hbase local files via ambari

Re: Hbase Storing pdf and Retrieval

Re: Does Ambari automatically adjust memory settin...

Re: HBase Regions stuck in FAILED_OPEN state

Re: Cannot start Ambari-metrics-collector

Re: DataFrames: inefficiencies of keeping all fiel...

Re: Caused by: java.lang.IllegalAccessError: tried...

Re: spark-shell error : No FileSystem for scheme: ...

Re: spark-shell error : No FileSystem for scheme: ...

Re: How do you test an HBase Endpoint Coprocessor?

Re: How do you test an HBase Endpoint Coprocessor?

Re: How can I be sure an HBase Endpoint co-process...

Re: How can I be sure an HBase Endpoint co-process...

Re: How can I be sure an HBase Endpoint co-process...

Re: The node /hbase is not in ZooKeeper. It should...

Re: phonenix jar client hbase insert

Re: Combine few hbase shell commands

Re: CBO for Hive over hbase

Re: Combine few hbase shell commands

Re: HBase regionserver is down during importtsv an...

Re: HBase regionserver is down during importtsv an...

Re: Unable to import data into hbase table through...

Re: Unable to import data into hbase table through...

Re: oldWALs folder growing too much....

Re: oldWALs folder growing too much....

Re: NiFi Phoenix processor?

Re: Unable to import data into hbase table through...

Re: Unable to import data into hbase table through...