Member since
09-29-2015
94
Posts
117
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
654 | 08-04-2017 06:13 PM | |
2246 | 03-21-2017 08:00 PM | |
463 | 11-30-2016 07:38 PM | |
334 | 11-03-2016 05:56 PM | |
657 | 10-11-2016 06:44 PM |
08-04-2017
06:13 PM
You can use Time To Live (TTL) attribute for the table for this. You can read about it here https://hbase.apache.org/book.html#ttl
... View more
07-06-2017
04:08 PM
129793ms is a very long time indeed. I would suspect that the kernel is blocking the process and not scheduling it for some reason. I would look into kernel messages (like dmsg, /var/log/messages) etc for clues.
... View more
05-12-2017
10:28 PM
You can do this: - Manually kill the server server05 by issuing kill -9. This will cause the master to recognize that the server is dead, and will re-assign the regions that were hosted there. Also you can safely restart the master in a production env. Nothing in Hbase client depends on master being available, in regular read / write paths (only DDL statements). Master is pretty light and will come up quikly, so you can restart masters safely.
... View more
04-06-2017
10:27 AM
1 Kudo
Maybe you don't have the patch. Which version of HDP is this? You can try something like this: export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.security.auth.login.config=/usr/hdp/current/hbase-client/conf/hbase_client_jaas.conf"
Then run the hbase shell command.
... View more
04-06-2017
08:15 AM
1 Kudo
The problem is that by default, when you launch hbase shell, it does not authenticate to zookeeper. For doing replication-related operations, you should be authenticating as the hbase server-side user. The easiest way, would be to launch the shell like this: hbase --auth-as-server shell See https://issues.apache.org/jira/browse/HBASE-15145
... View more
03-21-2017
08:00 PM
If you read the exception, it clearly says what is wrong: NoSuchColumnFamilyException:Column family id does not exist in You have to have a column family in the mapping of column id. Also, I'm not sure whether using >8 column families is the right design. Unless you really know that you need column familiies for the IO pattern, just stick to one. Enis
... View more
03-09-2017
07:28 PM
1 Kudo
How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment.
... View more
03-03-2017
10:47 PM
ZooKeeper has SSL with Netty. But I am not sure it is tested well. https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide @Josh, in case of HBase tokens, I think they are stored in ZK. Can this be a concern?
... View more
03-02-2017
07:42 PM
You should close the Connection / Admin / Table WHEN you are done. According to #3, Connection creation is very costly, so you would want to share the connection as much as possible, which means that you won't be "done" with the Connection until the application shuts down. Close the Connection when you know that you won't be doing any more requests. However, Table and Admin is relatively cheap, so you should open / close those per request.
... View more
03-02-2017
07:21 PM
What do you mean by "When HBase Coprocessor registers this added data"? Are you doing puts against another table, or creating a table? What is your coprocessor code doing exactly?
... View more
02-15-2017
07:30 PM
1 Kudo
Thanks Artem. This is still a little bit low level. You can use org.apache.hadoop.hbase.security.access.AccessControlClient as a client-facing API to grant / revoke at a higher level.
... View more
02-10-2017
10:12 PM
1 Kudo
Did you enable the backup configuration? It should be there in the HDP2-5. documentation. @mqureshi you can read this: http://hortonworks.com/blog/coming-hdp-2-5-incremental-backup-restore-apache-hbase-apache-phoenix/
... View more
12-15-2016
10:31 PM
2 Kudos
HBase in HDP is certified to run on top of HDFS, WASB/ADLS (using Azure HDInsight) and EMC Isilon.
... View more
12-15-2016
09:54 PM
4 Kudos
That is not a real error. ZK is just saying that it won't do SASL when connecting to the server (because this is not a secure cluster). You can ignore it.
... View more
12-15-2016
09:48 PM
Also, deploying the region observer like this is not the correct way to go. It will make it so that ALL tables in HBase are deployed with this coprocessor (including the system tables). Since they do not have the "cf1" column, operations will fail for system table and you cannot get a cluster up and running. Please use the table property for the coproccessor for selectively enabling the coprocessor only for the tables that you need. See https://hbase.apache.org/book.html#_guidelines_for_deploying_a_coprocessor
... View more
11-30-2016
07:38 PM
The best way to observe and run custom logic when a table is modified would be to write a master coprocessor. Your MasterObserver will be called before and after any table is modified (see https://hbase.apache.org/book.html#cp). However, unlike column families, columns in HBase are not predefined. So you cannot observe column changes since there is no fixed "schema". Please read up on the HBase data model.
... View more
11-29-2016
07:19 PM
1 Kudo
You should use the Hive -> Phoenix connector instead of the Hive -> HBase connector since the table is a Phoenix table. More information here: https://phoenix.apache.org/hive_storage_handler.html
... View more
11-15-2016
06:54 PM
You should have hbase-site.xml in your classpath for the java application. Please read: https://community.hortonworks.com/articles/4091/hbase-client-application-best-practices.html
... View more
11-08-2016
08:04 PM
Hbase in HDP-2.5 will NOT work with protobuf-3.0. Please refrain from manually changing libraries or classpath entries of any of the HDP components.
... View more
11-03-2016
05:56 PM
1 Kudo
TableInputFormat used in HBase will create 1 map task per table region. The data size will depend on how big your regions are.
... View more
10-17-2016
09:00 PM
5 Kudos
Most of the HBase features are cell-oriented rather than row-oriented unlike RDBMSs. For example the TTL is decided based on each individual cell, rather than a given row. Compactions (which is how HBase expires data) will also work for column families separately. They will never see the whole data for a given row. However, you can still implement what you want with some amount of code. As Josh suggests, you can actually implement a Filter that will only return rows that match your TTL criteria. Then you can issue deletes for those rows periodically.
... View more
10-11-2016
06:44 PM
1 Kudo
Scans in HBase work in batches since there is no streaming RPC in HBase. A scanner is opened for a region and the scan is executed as a series of RPC calls to fetch the next set of results. Every such call is a "next" operation, referred as ScanNext. The scan next call tries to fetch either a predefined set of rows (scanner caching) or predefined max result size (2MB, etc). The behavior depends on the version of HBase as well as configuration. More info here: https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1 Seeing 10-90 seconds in the latency metrics means that most of the RPC call to get the next scan results ended up taking that long. It maybe due to a case where the scan is scanning a lot of data with a highly selective filter and not returning data or something else is wrong causing excessive latency for the scans.
... View more
10-07-2016
08:56 PM
1 Kudo
The regionserver is failing because of this:
2016-10-0703:01:03,102 WARN [master/imp1tvhdpmst1.corp.test.com/172.24.125.130:16000] util.Sleeper:We slept 63817ms instead of 3000ms,thisis likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired Please check your GC settings and tune the GC.
... View more
10-05-2016
06:06 PM
1.6 is not supported. Please use JDK-1.8.
... View more
09-30-2016
06:28 PM
1 Kudo
Ambari does not use these files for managing the HBase service. Those files are only used for custom scripts that come with HBase (bin/start-hbase.sh, etc) which is not needed in an Ambari setup.
... View more
09-29-2016
10:31 PM
You cannot use the Scan's time range filters because as you have guessed HBase is not a row-oriented engine, but a cell-oriented one. The correct approach is to write a Filter which will decide whether to include the whole row or not. For doing that, you can set the timerange in your filter, and override Filter.filterRow() and filterKeyValue() methods and keep the state within the row, and decide to include the row or not based on the Cells matching the timerange or not. You can find example filters to look at in the source code or elsewhere.
... View more
09-27-2016
06:20 PM
1 Kudo
You are probably running into https://issues.apache.org/jira/browse/HBASE-1514. In secure clusters, you should run the zkcli command with --auth-as-server parameter. Like this: hbase --auth-as-server zkcli In newer versions of HDP, the patch should be there. Otherwise, you can get this working with something like: HBASE_OPTS="$HBASE_OPTS -Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_master_jaas.conf hbase zkcli Please note that the jaas.conf file that you add to the HBASE_OPTS before running the zkcli command should match what you have for your master and regionservers.
... View more
09-14-2016
09:03 PM
1 Kudo
HBase client running inside the mapper has to be able to connect to HBase with kerberos authentication. Normally if this was not a MR job, the client application is usually run within a login context where the keytabs are deployed and kinit has been run.
When the hbase client is run inside a task (mapper or reducer) in MR or in Spark, the client still has to authenticate. Since these tasks are run in a different node and a different process, the original TGT the application has obtained is obviously not available. TableInputFormat solves this case automatically by obtaining "delegation tokens" and distributing these delegation tokens together with the MR job. Luckily, these are already built in MR framework, so that token distribution will be taken care of.
You can call: TableMapReduceUtil.initCredentials(job)
before you submit your job, so that the Connection that you create in the Mapper side will automatically get the tokens and be able to connect to HBase. Notice that you should still kinit() or login with keytab in the application who submits the MR job.
... View more